[SAoC] "Improving DMD as a Library" project thread

[Jacob Carlborg]

On Friday, 30 October 2020 at 06:03:40 UTC, RazvanN wrote:

> So right now the compiler, when given a .d/.di file it opens 
> it, reads the contents and immediately lexes+parses the string 
> after which the string is discarded. If the contents of the 
> file need to be changed or reanalyzed, then the whole process 
> needs to be started from scratch. What you are proposing Jacob 
> is that the contents of the file are stored somewhere for ease 
> of reuse. Is that right?

Kind of, or at least that's one of the reasons. The main idea is 
to separate the reading of a file from lexing and parsing it. We 
introduce a file manager (like a cache). The compiler will first 
look in the file manager if the file content if available, 
otherwise read from disk. The important part here is that it 
needs to be possible to pre-populate (and also update) the file 
manager with a file and its content. This would allow to do a 
full compilation from memory, without touching the disk.

The main reason for this is to be able to have the compiler 
receive file content data from other sources than disk. Two use 
cases for that would be:

* A LSP server (or similar tool) receiving the data from the 
network from an editor with unsaved files

* The data is already in memory, think a string literal. This is 
useful when writing tests

The other idea is, as you mentioned, to read from memory if the 
file has already been read from disk when reanalyzing. For 
example, if you want to get the tokens of an AST node, as the 
compiler looks like now, you probably need to re-lex the file to 
get the tokens. But you don't want to re-read the file from disk, 
because it might have been updated. For this use case, it's 
really important the compiler is reading the exact same file 
content as it did when it originally created the AST.

Note, there's already a file cache [1], but that will not fit. It 
it's not possible to pre-populate or update. It also splits up 
the file in lines. The existing file cache [1] could perhaps take 
advantage of the new file manager.

Keep in mind that this new file manager needs to be used, not 
only when reading D files, but also when reading files through 
import expressions.

[1] https://github.com/dlang/dmd/blob/master/src/dmd/filecache.d

--
/Jacob Carlborg