Removing libdparse dependency from DLS

[Laurent Tréguier]

On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:
> I am working on DLS (https://github.com/d-language-server/dls), 
> trying to change the implementation to use the compiler as a 
> library. The reason behind this is to make DLS be always up to 
> date with the compiler so that it won't provide any obsolete 
> information.
>
> At the moment, libdparse 
> (https://github.com/dlang-community/libdparse) is used instead 
> and I want to get rid of this dependency. However, as DLS 
> doesn't really do much by itself, since it relies on other 
> tools, such as dcd (https://github.com/dlang-community/DCD), 
> dscanner (https://github.com/dlang-community/D-Scanner), dfmt 
> (https://github.com/dlang-community/dfmt), to do its work. 
> Unfortunately, these dependencies also use libdparse, so 
> removing it from DLS wouldn't be as easy as just adding the dmd 
> logic to the DLS code where libdparse is used.
>
> The cleanest way to solve this would be to replace libdparse 
> with dmd in dcd and dscanner, as well. Would this be a good 
> idea? Should I approach this differently?

As DLS' developer, I've looked into doing this myself a while 
back, close to a year ago now.
At that point I wasn't sure if it was really a good idea; even if 
DMD has technically been usable as a library for quite some time, 
in practice it wasn't as straightforward as I had hoped.

 From what I recall, the necessary changes to DMD would be:
- Removing any remaining global state. I don't know what the 
status of global state in DMD is nowadays. Although this might 
not be "necessary", it's always good to know that we can safely 
repeat calls to some APIs and always get the same output, without 
interference from some internal variables.
- Being able to fully control the standard input and output. When 
I tried to use DMD, there were some places where it was 
impossible to prevent DMD from using stdout IIRC. This is really 
critical, as standard input/output is the way language servers 
are used 99% of the time, and thus they need 110% control over 
stdin/stdout.
- Being able to free all the memory DMD allocates. From what I've 
read here, DMD doesn't free memory to get a higher compilation 
speed. However, language servers are long-running background 
programs, and any kind of memory leak is extremely bad in this 
situation. Since the apparition of the `-lowmem` flag, maybe this 
has improved.

Now, I haven't looked at all that since something like March of 
last year, so things might be better today (and they probably 
are, i think I recall Walter not being fond of DMD's global state 
either). I've gotten my hands busy with other projects, so 
development on DLS has basically halted and I don't expect to get 
back on it very soon.
My original plan wasn't to convert DCD/D-Scanner/DFMT, but to 
rewrite all functionality inside DLS from scratch using DMD. On 
the other hand, converting those tools would certainly benefit 
the whole community.

Rainers, VisualD's maintainer, has recently released a version of 
VisualD that can use DMD as a library; maybe its source code 
could provide you with some useful insight.

Good luck converting all those tools to use DMD, this is quite 
some work. If you do follow up on converting DLS, don't hesitate 
to open up issues in its repo if you have questions about how it 
works, I'll try my best to remember what I did and why!