[gsoc] D Language Server protocol based on the DMD frontend

[Seb]

Another thread on a project from the GSoC wiki to summarize all 
information, spark discussions and inspire students.

The excerpt of this project from the wiki:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A Language Server is meant to provide the language-specific 
smarts and communicate with development tools over a protocol 
that enables inter-process communication.
The Language Server Protocol [1] (LSP) defines the protocol used 
between an editor or IDE and a language server that provides 
language features like auto complete, go to definition, find all 
references etc.

Adding features like auto complete, go to definition, or 
documentation on hover for a programming language takes 
significant effort. Traditionally this work had to be repeated 
for each development tool, as each tool provides different APIs 
for implementing the same feature (see the list of Editors [2] 
and IDEs [3]). Additionally as the existing tools use `libdparse` 
(used in DLS [4] and DCD [5]), they lack the ability to perform 
semantic analysis on the AST and can't evaluate `mixin`'s nor D's 
very powerful Compile-Time Function Evaluation (CTFE).

The idea behind the Language Server Protocol (LSP) is to 
standardize the protocol for how such servers and development 
tools communicate. This way, a single D Language Server can be 
re-used in multiple development tools, which in turn can support 
multiple languages with minimal effort.

Recent development in the DMD compiler made it available as a DUB 
library [7]. This mean that the compiler itself can be re-used 
and third-party tools don't have to re-implement e.g. D's complex 
CTFE. This project is about using the DMD compiler as a library 
to implement a Language Server for D.

How to start:

* Simple example of using DMD as a library [8]
* More complex use cases [9]

[1] 
https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D
[2] https://wiki.dlang.org/Editors
[3] https://wiki.dlang.org/IDEs
[4] https://microsoft.github.io/language-server-protocol
[5] https://github.com/d-language-server/dls
[6] https://github.com/dlang-community/DCD
[7] 
https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library
[8] 
https://github.com/dlang/dmd/blob/master/test/dub_package/frontend.d
[9] https://github.com/jacob-carlborg/dlp

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I think the project itself contains of two parts:

1) Improving DMD as a Library

We are still actively working on this, so expect that you might 
need to dive into DMD and refactor things.
Here's a recent example:

https://github.com/dlang/dmd/pull/9350

2) Exposing the DMD as a language server

You probably can get a many inspirations here from the existing 
tools:

- https://github.com/d-language-server/dls
- https://github.com/dlang-community/DCD


How to start
------------

I think the most crucial step is to familiarize yourself with the 
DMD codebase. It's very non-trivial and I think the best way is 
to start fixing easier DMD bugs.
Often ICEs are good for beginners as they already point you to 
the rough location which most likely needs fixing.

A more concrete suggestion from Jacob
-------------------------------------

When it comes to improving DMD as a library, I think one of the 
key things would be to reduce global state. When the compiler is 
used as a library it needs to be able to invoke the library 
multiple times without changing the outcome. Not be dependent of 
previous state. We have a few workarounds , like [1], to reset 
the state, but it would be much better if there wasn’t any global 
state to begin with.

A related thing is that all diagnostic handling (after the 
parsing phase) goes through a global function which always 
outputs to stderr. This makes it basically impossible to 
configure the diagnostic handling which is useful when using the 
compiler as a library.

Using the `pure` function attribute helps to identify global 
state. My tool, DLP [2], allows to identify leaf functions 
(functions that don’t call other functions), which would be a 
good start to add attributes on. My tool can also serve as an 
example of using the DMD as a library.

Perhaps adding to DLP, or writing a new tool, that can suggest 
attributes that can be added to functions could be a good start. 
It would both help by learning how to use DMD as a library and 
the tool can then be used to identify which functions can be 
`pure` and which can’t.


[1] 
https://github.com/dlang/dmd/blob/6e2ee54aad00ff0f22dd1f9139ecea230aa8426a/src/dmd/frontend.d#L81-L108
[2] http://github.com/jacob-carlborg/dlp

Community
---------

@community: what language smarts in an editor are the most 
important to you? What features do you miss in DLS/DCD?

@students: please don't be shy. This is on open discussion and 
your questions are welcome!