How can we make it easier to experiment with the compiler?

[Johan Engelen]

On Monday, 24 May 2021 at 09:53:42 UTC, Walter Bright wrote:
> On 5/23/2021 10:55 PM, Andrei Alexandrescu wrote:
>> One problem with that is code duplication.
> Sure. But in the outbuffer case, the duplication stems from 
> backend being used in multiple projects.
>
> It's hard to have perfection, and if getting to perfection 
> means driving off a cliff (!) it's better to just live with a 
> bit of imperfection here and there.
>
> I don't like having two outbuffers, but the cure is worse for 
> that particular case.
>
> There's even another implementation of outbuffer in Phobos 
> (because I thought outbuffer was generally very useful):
>
> https://dlang.org/phobos/std_outbuffer.html
>
> But here we run into our rule that dmd shouldn't rely on 
> Phobos. Compromise is inevitable. Outbuffer isn't a spike we 
> need to impale ourselves on. There are plenty of other 
> encapsulation problems that could be improved, like target.d.

Outbuffer is a case of a data structure that is useful throughout 
the compiler. So it is put in a package of the compiler that is 
OK to be imported from other packages (and should avoid importing 
other packages). I think the `dmd.root` package is exactly like 
such a package  (compare with `ADT` in LLVM). From that 
standpoint, I don't see why the `dmd.backend` package cannot 
import `dmd.root`. If `dmd.backend` is to be used in different 
projects, then those should also use `dmd.root` and that's where 
the dependency chain stops.
Better: if it is in Phobos, great, use that!
If you need a certain data structure you know where to look: 
Phobos or `dmd.root`. Is it not in there? Don't create a new 
structure elsewhere, add it to `dmd.root` and import it.

In LDC, we use the C++ stdlib and we use Phobos, because why not? 
We are programming in D after all, and it is the standard library 
that is available in all bootstrapping compiler required 
versions. We do take care not to rely on _latest_ Phobos, but on 
Phobos from the oldest bootstrapping D compiler version supported 
up to the latest version. Same for the C++ stdlib (C++20 is not 
ok, but C++14 is much encouraged).

For example: LDC uses MD5 hashing for its machine codegen cache. 
`import std.digest.md; auto md5hash = md5Of(slice);`  Done.
LLVM does the same, e.g. the project removed its own unique_ptr 
implementation (`OwningPtr`), and now uses `std::unique_ptr`.


My standpoint on the original topic of "make it easier to 
experiment with the compiler": I disagree with making the code 
more stable. If anything, we should be refactoring much more 
aggressively, changing function names etc. Nicholas mentions that 
it is a pain to keep up with LLVM, where even function names are 
renamed from "SortSomeName" to "SortSomeNames" (made-up example), 
because plural is correct. The pain would be _much_ more if all 
unfixed small incorrectness/clumsiness/etc. accumulates over time 
and you end up with a convoluted codebase...
The main stumbling block is already mentioned: ownership. If you 
want contributors, you have to give up some ownership and be 
willing to make compromises between your own and the new 
contributors' viewpoints. The lack of willingness to give up 
ownership is what keeps me out (and I suspect, indirectly, others 
too). The frontend source code is not nice, but I'm not drawn to 
fix it at all (even if paid for) because I am not ashamed by it 
as I would be if I would have some shared 'ownership' of it.

-Johan