DMD as a library - recap, next steps

[RazvanN]

On Tuesday, 16 June 2020 at 09:00:55 UTC, Petar Kirov 
[ZombineDev] wrote:
> On Tuesday, 16 June 2020 at 08:16:25 UTC, Stefan Koch wrote:
>> [...]

> It is useful for serialization and source generation purposes 
> at least. In other language communities, where they don't have 
> the metaprogramming power of D, they do extensive source code 
> generation, and most of the time you don't need much semantic 
> for that (sometimes you just need a way to string interpolation 
> on symbol names).
>

Indeed!

> Also perhaps one of the goal is to completely replace the 
> parser with something more fault-tolerant. For example see:
>
> https://unallocated.com/blog/incremental-packrat-parsing-the-secret-to-fast-language-servers/
>
> https://github.com/tree-sitter/tree-sitter
> https://github.com/tree-sitter/tree-sitter/blob/master/docs/index.md
>
> ^^^
> I suggest you watch the talks linked from the last page.
>

That is exactly what I had in mind.

>> [...]
>
> I agree. Though if a refactoring of this sort increases the 
> compile-time by 2x we have serious problems elsewhere. I don't 
> expect something like this to require more 1.15x in the worst 
> case. And even then we should be able to find many ways to 
> further decrease the compile-time to e.g. 0.85x.
>

You are correct. I was exaggerating for the sake of the argument. 
What I meant was: we are extremely fast, but we are lacking a 
compiler interface. Small performance regressions are acceptable 
if they offer a guaranteed benefit with regards to defining a 
good interface.

>
> I suggest you try using a language with an excellent 
> implementation of a LSP (language server) like C# or 
> TypeScript. You'd be amazed how well it works. It is able to 
> make sense of all kinds of broken code (e.g. giving you 
> auto-completion for a function with a missing closing curly 
> brace. Most of these things are completely impossible to do 
> with the current rigid nature of the dmd frontend.
>
> Of course Razvan, Edi and Cristian are in better position to 
> answer, but I think that the main idea is that they don't want 
> to change the language, but instead they want to be able to 
> plug code in more parts of the compilation pipeline so the LSP 
> can be notified when e.g. the compiler is visiting an overload 
> set. For example, the compiler may stop looking when it finds 
> the best overload match, while for the LSP you want to display 
> overloads, as the user may have made a typo and so on. (Please 
> don't read too much into this example, I may have made a 
> mistake, but the general idea is that they need to be able to 
> extract more info from the frontend.)
>

You are entirely right. We have replaced all uses of libdparse and
other tools that mimic semantic analysis in DCD with dmd as a lib 
and
it works, but it requires that we override current semantic 
analysis that
is done for CallExps to be able to cope with semantic failures on 
incomplete code.

>> [...]
>
> The use case implementing all of this API:
>
> https://microsoft.github.io/language-server-protocol/specifications/specification-current/
>
> Specifically, take a look at the "Language Features" section, 
> e.g.:
> https://microsoft.github.io/language-server-protocol/specifications/specification-current/#textDocument_completion
>
>> [...]
>
> The set of C developers that want to write a D langauge server 
> is much smaller than the set of D developers that want to do 
> the same :D
>
> But I agree on the general point that a well-defined and 
> versioned API is much needed, just like it sucks when changes 
> frontend break LDC and GDC.

Thanks for your reply. I feel that we are on the same page on 
this.