DMD as a library - recap, next steps

[WebFreak001]

On Tuesday, 16 June 2020 at 04:13:10 UTC, RazvanN wrote:
> A few years ago, I worked on a project to refactor the dmd 
> codebase so that it becomes easier to be used as a library. 
> This has the advantage that tools that use dmd-as-a-lib will 
> rely on the latest working compiler. From that perspective, I 
> made several PRs:
>
> [...]
>
> Me and Edi are in the position where we can use bachelor 
> students to do the heavy-lifting on helping this project to 
> cross the finish line, however, it is critical that the Dlang 
> Foundation leadership has a clear direction/vision on this.
>
> So, how do we move forward?
>
> Cheers,
> RazvanN

very cool, thanks for all the work! I think however increasing 
the compilation time significantly is a major no-go, at least 
when it affects all code, expect of course if it allows us to do 
faster things like incremental compilation.

For architecture ideas you might want to check out how Microsoft 
implemented their Roslyn compiler platform: 
https://github.com/dotnet/roslyn/wiki/Roslyn-Overview

Just AST and visitors is not the most useful on its own, we have 
libdparse for this already which works fine too. Much more 
interesting is the semantic analysis.

If there is one thing I would want exposed by dmd for anything, 
being completion, dynamic linting, navigation, etc., I would 
really really want a symbol API. Much like dsymbol one 
incremental database of all defined symbols (modules, types, 
aliases, parameters, template parameters, variables, etc.) with 
references, definitions, types (of variables and parameters), 
names and all traits information. This database would contain all 
symbols in the entire compilation unit, be aware of scopes at any 
given point and be able to incrementally update by 
adding/removing or changing files.

The semantic analysis needs to be incremental here too though, so 
symbols would need some kind of dependency graph for things using 
mixin or templates. Also it would be difficult to extract 
information from scopes like `version (Foo)` where version is not 
Foo.

But just this symbols database and some APIs to query it per 
location, per file or per symbol name would be enough to 
implement nearly all features a user would expect from tooling 
and more.

Otherwise raw token access and AST visitors is all you really 
need to implement the rest like formatting, highlighting, static 
linting and refactorings. It's important that you can somehow 
recover the whitespaces and comments from tokens for refactoring 
and formatting though!

For other use-cases, like a REPL, exposing APIs to the executable 
generator would also be cool.

So if we have a symbols API I'm happy and I think that will be 
the goal of any DCD replacement program too :p

Keep up the good work on this!