DMD as a library - recap, next steps

[Petar]

On Tuesday, 16 June 2020 at 08:16:25 UTC, Stefan Koch wrote:
> On Tuesday, 16 June 2020 at 04:13:10 UTC, RazvanN wrote:
>
>> - Further strip the AST nodes of functions that require 
>> semantic analysis so that we can remove the code duplication 
>> in ASTBase
> The AST without SemA is useless.

It is useful for serialization and source generation purposes at 
least. In other language communities, where they don't have the 
metaprogramming power of D, they do extensive source code 
generation, and most of the time you don't need much semantic for 
that (sometimes you just need a way to string interpolation on 
symbol names).

Also perhaps one of the goal is to completely replace the parser 
with something more fault-tolerant. For example see:

https://unallocated.com/blog/incremental-packrat-parsing-the-secret-to-fast-language-servers/

https://github.com/tree-sitter/tree-sitter
https://github.com/tree-sitter/tree-sitter/blob/master/docs/index.md

^^^
I suggest you watch the talks linked from the last page.

>
>> - Refactor dmd to offer a decent interface. Sometimes people 
>> argue against this point by saying that some changes affect 
>> the compilation time of the compiler. Honestly, it takes 5 
>> seconds to compile dmd, even if we double that time I say it 
>> is worth it if we gain a decent interface for dmd-as-a-lib.
>
> No if we double that time, it's a massive change to the rapid
> development we can have right now.
> Try compiling DMD with LDC and you'll see what I mean

I agree. Though if a refactoring of this sort increases the 
compile-time by 2x we have serious problems elsewhere. I don't 
expect something like this to require more 1.15x in the worst 
case. And even then we should be able to find many ways to 
further decrease the compile-time to e.g. 0.85x.

>
>> In the mean time there were some PRs that regressed the state 
>> of dmd as a lib, PRs such as this one: 
>> https://github.com/dlang/dmd/pull/9010 . That PR makes it 
>> impossible for someone to override the type semantic.
>
> Why would you ever need or want to OVERRIDE semantic.
> If you override semantics than by definition you are no longer,
> in the same language space.

I suggest you try using a language with an excellent 
implementation of a LSP (language server) like C# or TypeScript. 
You'd be amazed how well it works. It is able to make sense of 
all kinds of broken code (e.g. giving you auto-completion for a 
function with a missing closing curly brace. Most of these things 
are completely impossible to do with the current rigid nature of 
the dmd frontend.

Of course Razvan, Edi and Cristian are in better position to 
answer, but I think that the main idea is that they don't want to 
change the language, but instead they want to be able to plug 
code in more parts of the compilation pipeline so the LSP can be 
notified when e.g. the compiler is visiting an overload set. For 
example, the compiler may stop looking when it finds the best 
overload match, while for the LSP you want to display overloads, 
as the user may have made a typo and so on. (Please don't read 
too much into this example, I may have made a mistake, but the 
general idea is that they need to be able to extract more info 
from the frontend.)

>> Me and Edi are in the position where we can use bachelor 
>> students to do the heavy-lifting on helping this project to 
>> cross the finish line, however, it is critical that the Dlang 
>> Foundation leadership has a clear direction/vision on this.
>>
>> So, how do we move forward?
>
> First establish a usecase.
> I would say that the ability to custom add static-analysis 
> passes should be at the forefront, since that is what I would 
> want a compiler as a library to do.

The use case implementing all of this API:

https://microsoft.github.io/language-server-protocol/specifications/specification-current/

Specifically, take a look at the "Language Features" section, 
e.g.:
https://microsoft.github.io/language-server-protocol/specifications/specification-current/#textDocument_completion

> For Example, I want to be able to ask.
>
> Forall functions in module a, give me the ones which call a 
> function called malloc,
> either directly or transitively, as far as you can see by the 
> source code I gave you.
> Forall functions which only exist as declarations (you don't 
> have the body), create a list and cross reference it with the 
> call-graph of the selection we've got before.
>
> For that to work you need to be able to run time compiler until 
> just before code-generation and you need to be able to walk 
> that type/identifer-resolved tree in a useful manner.
>
> Some kind of C plugin api would be preferred.

The set of C developers that want to write a D langauge server is 
much smaller than the set of D developers that want to do the 
same :D

But I agree on the general point that a well-defined and 
versioned API is much needed, just like it sucks when changes 
frontend break LDC and GDC.