Migrating dmd to D?

[Adam Wilson]

On Wed, 27 Feb 2013 16:37:50 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail at erdani.org> wrote:

> Hello,
>
>
> Walter and I have had a long conversation about the next radical thing  
> to do to improve D's standing. Like others in this community, we believe  
> it's a good time to consider bootstrapping the compiler. Having the D  
> compiler written in D has quite a few advantages, among which taking  
> advantages of D's features and having a large codebase that would be its  
> own test harness.
>
> By this we'd like to initiate a dialog about how this large project can  
> be initiated and driven through completion. Our initial basic ideas are:
>
> 1. Implement the dtoh standalone program that takes a D module and  
> generates its corresponding C++ header.
>
> 2. Use dtoh to initiate and conduct an incremental port of the compiler.  
> At given points throughout the code D code will coexist and link with  
> C++ code.
>
> 3. At a point in the future the last C++ module will be replaced with a  
> D module. Going forward there will be no more need for a C++ compiler to  
> build the compiler (except as a bootstrapping test).
>
> It is essential that we get support from the larger community for this.  
> This is a large project that should enjoy strong leadership apart from  
> Walter himself (as he is busy with dynamic library support which is  
> strategic) and robust participation from many of us.
>
> Please chime in with ideas on how to make this happen.
>
>
> Thanks,
>
> Andrei

First off, I am totally in favor of rewriting D in D. However, we should  
move carefully as there is a minefield of potential issues here.

The most important benefit of this project that I see is that it would  
force the codebase to rapidly stabilize by forcing the developers to make  
the language and library work for the compiller.

Andrei, specifically, having worked in the DMD code, it is, at this time,  
somewhat unrealistic to expect to be able port the code one file at a  
time. As has been previously discussed, DMD makes use of C++ features that  
D can't link with and in many cases the current code is not well  
modularized. The DMD C++ code would have to be very carefully refactored  
with D porting in mind PRIOR to beginning the actual porting project. That  
will incur an additional time penalty.

I don't see this happening without a project freeze. We absolutely cannot  
be porting code and fixing bugs at the same time. Even if we did one  
file/module at a time, many of the files are multiple thousands of lines  
of code. Just doing a straight conversion of a single file will take  
non-trivial amounts of time.

The legal issues surrounding the Back-End have been the cause of great  
concern among many of the Linux distro packagers. This is a VERY grave  
issue that MUST be addressed. One of the prime reasons driving the  
ubiquity of GCC is that it's license is acceptable to virtually every  
distro available. DMD will NEVER be available on distro's like Debian.  
THAT is a PROBLEM.

My preference would be to completely replace the back-end with LLVM. Why  
LLVM? Well as opposed to GCC it was designed from the ground up to support  
many languages. The benefit here is that it is possible to create  
standalone compiler the generates LLVM bytecode that can then be run  
through LLVM. My understanding (and I am happy to be corrected here) is  
that LLVM does not need the front-end to be compiled into the back-end.

I would ask the community and particularly Walter to consider the  
following plan:
Freeze the DMD repo.
Set-up a new repo for the DMD in D codebase.
Walter creates the basic folder/file structure.
The community can then begin porting the compiler by submitting pull  
requests, this allows multiple people to potentially port the same code  
and the core team can select the best conversion.
Switch the backend to LLVM. This would reap the following benefits.
	A highly optimized code generator. They have hundreds of people working  
on theirs. We don't.
	Reduction in specialized knowledge with DMD. Very few people understand  
the DMDBE. The bus factor is uncomfortably high.
	Reduction in workload for core team. By removing the need to support the  
backend the team will be able to focus on the front-end.
	Portability. Simply put, the amount of work required to make DMD work on  
ARM is beyond reasonable and ARM support is absolutely required in the  
future of computing. If we used LLVM this becomes almost trivially easy.  
Just rework druntime/phobos for ARM.
Once the port is complete and working we unfreeze DMD for bug fixes and  
get back to it.

I suspect that by allowing the many people skilled in D to port the code  
in a simultaneous fashion that we could complete the porting in a matter  
of months. It would be longer than the desired release cycle of two  
months. But I would think that four months is a reasonable estimate.

Ok D community, destroy me!

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/