DMD as a library - recap, next steps

[RazvanN]

On Tuesday, 16 June 2020 at 10:48:13 UTC, WebFreak001 wrote:

>
> very cool, thanks for all the work! I think however increasing 
> the compilation time significantly is a major no-go, at least 
> when it affects all code, expect of course if it allows us to 
> do faster things like incremental compilation.
>
Incremental compilation is out of discussion when referring to 
dmd as a lib. For that we would need to start from scratch an 
implementation that takes care of all the various cases.

> For architecture ideas you might want to check out how 
> Microsoft implemented their Roslyn compiler platform: 
> https://github.com/dotnet/roslyn/wiki/Roslyn-Overview
>
> Just AST and visitors is not the most useful on its own, we 
> have libdparse for this already which works fine too. Much more 
> interesting is the semantic analysis.
>
But the whole point of dmd as a lib is to offer the ability to 
use semantic analysis by using semantic visitors. For example 
this PR
https://github.com/dlang/dmd/pull/11265 enables the ability to 
inherit semantic visitors that are used in the compiler and 
override or extend the functionality.

> If there is one thing I would want exposed by dmd for anything, 
> being completion, dynamic linting, navigation, etc., I would 
> really really want a symbol API. Much like dsymbol one 
> incremental database of all defined symbols (modules, types, 
> aliases, parameters, template parameters, variables, etc.) with 
> references, definitions, types (of variables and parameters), 
> names and all traits information. This database would contain 
> all symbols in the entire compilation unit, be aware of scopes 
> at any given point and be able to incrementally update by 
> adding/removing or changing files.
>

One step in that direction: 
https://github.com/dlang/dmd/pull/11092
With that, you can provide a function that pulls out all the 
symbols in a particular scope. It is not incremental, though.

> The semantic analysis needs to be incremental here too though, 
> so symbols would need some kind of dependency graph for things 
> using mixin or templates. Also it would be difficult to extract 
> information from scopes like `version (Foo)` where version is 
> not Foo.
>
> But just this symbols database and some APIs to query it per 
> location, per file or per symbol name would be enough to 
> implement nearly all features a user would expect from tooling 
> and more.
>
> Otherwise raw token access and AST visitors is all you really 
> need to implement the rest like formatting, highlighting, 
> static linting and refactorings. It's important that you can 
> somehow recover the whitespaces and comments from tokens for 
> refactoring and formatting though!
>
> For other use-cases, like a REPL, exposing APIs to the 
> executable generator would also be cool.
>
> So if we have a symbols API I'm happy and I think that will be 
> the goal of any DCD replacement program too :p
>
We are working on this and will soon publish that work.

> Keep up the good work on this!