DMD as a library - recap, next steps

[Jacob Carlborg]

On Tuesday, 16 June 2020 at 04:13:10 UTC, RazvanN wrote:
> A few years ago, I worked on a project to refactor the dmd 
> codebase so that it becomes easier to be used as a library. 
> This has the advantage that tools that use dmd-as-a-lib will 
> rely on the latest working compiler. From that perspective, I 
> made several PRs:
>
> What remains to be done is to:
>- Refactor dmd to offer a decent interface. Sometimes people
> argue against this point by saying that some changes affect the 
> compilation time of the compiler. Honestly, it takes 5 seconds 
> to compile dmd, even if we double that time I say it is worth 
> it if we gain a decent interface for dmd-as-a-lib. On this 
> point, me and Edi Staniloiu have been working with a bachelor 
> student to see what interface is required to be able to use 
> dmd-as-a-lib in tools in the ecosystem (like DCD).
> - Add some visitors that make it easy for 3rd party tools to 
> use compiler features.

I don't think API is not the most important part, as long as you 
can do what you need to do with the library. In my opinion 
there's a lot of functionality that is missing. For example, I've 
tried to use DMD as a library to do source code transformation. 
It falls very short in this area:

* AST nodes without locations
* Locations don't contain an end point/length
* Locations don't contain the buffer offset
* Indirect files are always read from disk. There's no option to 
make a full compilation purely from memory

> Me and Edi are in the position where we can use bachelor 
> students to do the heavy-lifting on helping this project to 
> cross the finish line, however, it is critical that the Dlang 
> Foundation leadership has a clear direction/vision on this.

I agree, you need a full buy in from the leadership and the 
compiler developers. I think this will be very difficult.

> So, how do we move forward?

I think the way forward is to fork DMD to allow you to make the 
necessary changes as you see fit without having to bother with 
discussion and politics.

If you're lucky you can merge changes from upstream easily. 
Otherwise if you can't easily sync with upstream you can treat it 
as a separate compiler and evolve it on its own.

I've already done this, if you're interested in collaborating [1].

[1] https://github.com/jacob-carlborg/ddc

--
/Jacob Carlborg