DMD as a library - recap, next steps

[RazvanN]

On Tuesday, 16 June 2020 at 08:16:25 UTC, Stefan Koch wrote:
> On Tuesday, 16 June 2020 at 04:13:10 UTC, RazvanN wrote:
>
>> - Further strip the AST nodes of functions that require 
>> semantic analysis so that we can remove the code duplication 
>> in ASTBase
> The AST without SemA is useless.
>
It is not useless, the fact that libdparse exists and it is used 
as a standalone library is proof of that.
>
>> - Refactor dmd to offer a decent interface. Sometimes people 
>> argue against this point by saying that some changes affect 
>> the compilation time of the compiler. Honestly, it takes 5 
>> seconds to compile dmd, even if we double that time I say it 
>> is worth it if we gain a decent interface for dmd-as-a-lib.
>
> No if we double that time, it's a massive change to the rapid
> development we can have right now.
> Try compiling DMD with LDC and you'll see what I mean
>
The point here was that minor performance regressions that come 
out from refactorings should not be an obstacle if it offers a 
clear benefit from a dmd-as-a-lib standpoint.

>> In the mean time there were some PRs that regressed the state 
>> of dmd as a lib, PRs such as this one: 
>> https://github.com/dlang/dmd/pull/9010 . That PR makes it 
>> impossible for someone to override the type semantic.
>
> Why would you ever need or want to OVERRIDE semantic.
> If you override semantics than by definition you are no longer,
> in the same language space.
>
>> Me and Edi are in the position where we can use bachelor 
>> students to do the heavy-lifting on helping this project to 
>> cross the finish line, however, it is critical that the Dlang 
>> Foundation leadership has a clear direction/vision on this.
>>
>> So, how do we move forward?
>
> First establish a usecase.
> I would say that the ability to custom add static-analysis 
> passes should be at the forefront, since that is what I would 
> want a compiler as a library to do.
>
> For Example, I want to be able to ask.
>
> Forall functions in module a, give me the ones which call a 
> function called malloc,
> either directly or transitively, as far as you can see by the 
> source code I gave you.
> Forall functions which only exist as declarations (you don't 
> have the body), create a list and cross reference it with the 
> call-graph of the selection we've got before.
>
> For that to work you need to be able to run time compiler until 
> just before code-generation and you need to be able to walk 
> that type/identifer-resolved tree in a useful manner.
>
Analyzing the AST is one scenario, but there are other situations:

1. You want to extend the language with some feature
2. Semantic analysis mutates the AST in a way that makes it 
impossible for you to reason about what was there in the first 
place. One example here is an auto-complete tool that needs to be 
able to analyze incomplete code; if you do not override specific 
semantic methods, by the time you analyze the AST you will have 
error nodes that prevent you from doing any work.
3. You want to drop certain semantic passes because your tool 
does not necessitate them.

Ideally we would offer maximum flexibility with dmd-as-a-lib. 
Currently, you are forced to run the full semantic analysis pass 
and hope for the best.

> Some kind of C plugin api would be preferred.

I don't understand what you are reffering to.