DMD as a library - recap, next steps

[Jacob Carlborg]

On 2020-06-17 07:08, RazvanN wrote:

> This typically happens when the compiler rewrites segments of the AST. 
> It creates nodes, but doesn't bother with location since that code is 
> not meant to be seen by any user.

I wasn't referring to those cases. There are other cases I was thinking of:

@("bar")
void foo();

In the above code, the first location will point to, IIRC, the quote 
symbol (") and not the at sign (@). This is before running any semantic 
analysis.

> There are all valid points. I mostly thought about dmd as a lib as a way 
> to analyze the AST and output relevant information (e.g. DCD), not as a 
> tool to modify source code, however,

I don't know how far you've come with this progress and I'm certainly 
not an expert in this subject and I've only glanced at the LSP 
specification. But if the compiler cannot compile all files from memory 
the LSP server needs to store the source code in temporary files, or it 
needs to read the files directly from the project directory, the latter 
is what DCD is doing. In that case all files, perhaps except the one 
you're currently are editing in need to be saved. That is, you cannot 
have multiple unsaved files and get the correct result, you'll get stale 
data instead. Perhaps not that common. But you can definitely end up 
with multiple unsaved files after a global search-and-replace.

When it comes to start and end position and buffer offset, keep in mind 
that LSP does support modifying the source code with the "rename" 
feature [1]. If you don't know the buffer offset of a token, how would 
you know where to make the changes in the buffer? In this case you might 
get away with what the current compiler supports because this feature 
only applies to identifiers and you do know the length of an identifier. 
But don't you need to know where in the buffer the identifier start? I 
guess you could run the lexer again and count the number of bytes. But 
seems quite inefficient for something the compiler should support out of 
the box.

Another feature which seems to depend on start and end position, and 
possible buffer offset as well, is the "foldingRange" feature [2].

> I was expecting that the hdrgen  visitor would help with that.

I haven't looked at the implementation of hdrgen but if the lexer 
doesn't preserve the information how would hdrgen get access to it?

> I am still hoping that we can work our way with the main compiler

Yeah, I don't want to wait anymore.


[1] 
https://microsoft.github.io/language-server-protocol/specifications/specification-3-14/#textDocument_rename

[2] 
https://microsoft.github.io/language-server-protocol/specifications/specification-3-14/#textDocument_foldingRange

-- 
/Jacob Carlborg