[gsoc] D Language Server protocol based on the DMD frontend

[Rubn]

On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:
> Another thread on a project from the GSoC wiki to summarize all 
> information, spark discussions and inspire students.
>
> The excerpt of this project from the wiki:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> A Language Server is meant to provide the language-specific 
> smarts and communicate with development tools over a protocol 
> that enables inter-process communication.
> The Language Server Protocol [1] (LSP) defines the protocol 
> used between an editor or IDE and a language server that 
> provides language features like auto complete, go to 
> definition, find all references etc.
>
> Adding features like auto complete, go to definition, or 
> documentation on hover for a programming language takes 
> significant effort. Traditionally this work had to be repeated 
> for each development tool, as each tool provides different APIs 
> for implementing the same feature (see the list of Editors [2] 
> and IDEs [3]). Additionally as the existing tools use 
> `libdparse` (used in DLS [4] and DCD [5]), they lack the 
> ability to perform semantic analysis on the AST and can't 
> evaluate `mixin`'s nor D's very powerful Compile-Time Function 
> Evaluation (CTFE).
>
> The idea behind the Language Server Protocol (LSP) is to 
> standardize the protocol for how such servers and development 
> tools communicate. This way, a single D Language Server can be 
> re-used in multiple development tools, which in turn can 
> support multiple languages with minimal effort.
>
> Recent development in the DMD compiler made it available as a 
> DUB library [7]. This mean that the compiler itself can be 
> re-used and third-party tools don't have to re-implement e.g. 
> D's complex CTFE. This project is about using the DMD compiler 
> as a library to implement a Language Server for D.
>
> How to start:
>
> * Simple example of using DMD as a library [8]
> * More complex use cases [9]
>
> [1] 
> https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D
> [2] https://wiki.dlang.org/Editors
> [3] https://wiki.dlang.org/IDEs
> [4] https://microsoft.github.io/language-server-protocol
> [5] https://github.com/d-language-server/dls
> [6] https://github.com/dlang-community/DCD
> [7] 
> https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library
> [8] 
> https://github.com/dlang/dmd/blob/master/test/dub_package/frontend.d
> [9] https://github.com/jacob-carlborg/dlp
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> I think the project itself contains of two parts:
>
> 1) Improving DMD as a Library
>
> We are still actively working on this, so expect that you might 
> need to dive into DMD and refactor things.
> Here's a recent example:
>
> https://github.com/dlang/dmd/pull/9350
>
> 2) Exposing the DMD as a language server
>
> You probably can get a many inspirations here from the existing 
> tools:
>
> - https://github.com/d-language-server/dls
> - https://github.com/dlang-community/DCD
>
>
> How to start
> ------------
>
> I think the most crucial step is to familiarize yourself with 
> the DMD codebase. It's very non-trivial and I think the best 
> way is to start fixing easier DMD bugs.
> Often ICEs are good for beginners as they already point you to 
> the rough location which most likely needs fixing.
>
> A more concrete suggestion from Jacob
> -------------------------------------
>
> When it comes to improving DMD as a library, I think one of the 
> key things would be to reduce global state. When the compiler 
> is used as a library it needs to be able to invoke the library 
> multiple times without changing the outcome. Not be dependent 
> of previous state. We have a few workarounds , like [1], to 
> reset the state, but it would be much better if there wasn’t 
> any global state to begin with.
>
> A related thing is that all diagnostic handling (after the 
> parsing phase) goes through a global function which always 
> outputs to stderr. This makes it basically impossible to 
> configure the diagnostic handling which is useful when using 
> the compiler as a library.
>
> Using the `pure` function attribute helps to identify global 
> state. My tool, DLP [2], allows to identify leaf functions 
> (functions that don’t call other functions), which would be a 
> good start to add attributes on. My tool can also serve as an 
> example of using the DMD as a library.
>
> Perhaps adding to DLP, or writing a new tool, that can suggest 
> attributes that can be added to functions could be a good 
> start. It would both help by learning how to use DMD as a 
> library and the tool can then be used to identify which 
> functions can be `pure` and which can’t.
>
>
> [1] 
> https://github.com/dlang/dmd/blob/6e2ee54aad00ff0f22dd1f9139ecea230aa8426a/src/dmd/frontend.d#L81-L108
> [2] http://github.com/jacob-carlborg/dlp
>
> Community
> ---------
>
> @community: what language smarts in an editor are the most 
> important to you? What features do you miss in DLS/DCD?
>
> @students: please don't be shy. This is on open discussion and 
> your questions are welcome!

Using the actual DMD compiler may not be that great as a language 
server.

https://code.visualstudio.com/api/language-extensions/language-server-extension-guide#error-tolerant-parser-for-language-server

https://github.com/Microsoft/tolerant-php-parser/blob/master/docs/HowItWorks.md

It might be good to provide auto complete that is aware of things 
like mixins, but if you want language features when you are 
working on an incomplete incorrect source file. Not sure how good 
DMD will be able to handle that.

Whatever happened to the funding for code-d, did nothing ever 
come out of that?