DMD as a library - recap, next steps

[RazvanN]

A few years ago, I worked on a project to refactor the dmd 
codebase so that it becomes easier to be used as a library. This 
has the advantage that tools that use dmd-as-a-lib will rely on 
the latest working compiler. From that perspective, I made 
several PRs:

1. Template the parser to remove reliance on modules that 
implement semantic analysis: 
https://github.com/dlang/dmd/pull/6625

2. Create ASTBase, an AST family that contains the minimum 
information to separate the parser from the rest of the compiler: 
https://github.com/dlang/dmd/pull/6836

3. A series of PRs to pull out all the semantic methods from AST 
nodes into visitors or free functions so that AST nodes in the 
compiler will replace the ASTBase family (which is essentially 
duplicated code):
     https://github.com/dlang/dmd/pull/7031
     https://github.com/dlang/dmd/pull/7048
     https://github.com/dlang/dmd/pull/7049
     https://github.com/dlang/dmd/pull/7114
     https://github.com/dlang/dmd/pull/7119
     https://github.com/dlang/dmd/pull/7122

4. I created visitors for semantic time analysis: 
https://github.com/dlang/dmd/pull/7411

At that point I've hit some bugs that prevented me from moving 
forward. I started fixing them and after a while I moved to 
another project which left the dmd as a library project in a half 
baked state.

What remains to be done is to:

- Further strip the AST nodes of functions that require semantic 
analysis so that we can remove the code duplication in ASTBase
- Refactor dmd to offer a decent interface. Sometimes people 
argue against this point by saying that some changes affect the 
compilation time of the compiler. Honestly, it takes 5 seconds to 
compile dmd, even if we double that time I say it is worth it if 
we gain a decent interface for dmd-as-a-lib. On this point, me 
and Edi Staniloiu have been working with a bachelor student to 
see what interface is required to be able to use dmd-as-a-lib in 
tools in the ecosystem (like DCD).
- Add some visitors that make it easy for 3rd party tools to use 
compiler features.

In the mean time there were some PRs that regressed the state of 
dmd as a lib, PRs such as this one: 
https://github.com/dlang/dmd/pull/9010 . That PR makes it 
impossible for someone to override the type semantic. This is 
moving things in the exact opposite direction of every PR that 
was showcased on this post and is a showstopper for this: 
https://github.com/dlang/dmd/pull/11265 moving forward in a 
consistent way.

Me and Edi are in the position where we can use bachelor students 
to do the heavy-lifting on helping this project to cross the 
finish line, however, it is critical that the Dlang Foundation 
leadership has a clear direction/vision on this.

So, how do we move forward?

Cheers,
RazvanN