How can we make it easier to experiment with the compiler?

[Andrei Alexandrescu]

On 5/24/21 6:47 AM, Johan Engelen wrote:
> On Monday, 24 May 2021 at 09:53:42 UTC, Walter Bright wrote:
>> On 5/23/2021 10:55 PM, Andrei Alexandrescu wrote:
>>> One problem with that is code duplication.
>> Sure. But in the outbuffer case, the duplication stems from backend 
>> being used in multiple projects.
>>
>> It's hard to have perfection, and if getting to perfection means 
>> driving off a cliff (!) it's better to just live with a bit of 
>> imperfection here and there.
>>
>> I don't like having two outbuffers, but the cure is worse for that 
>> particular case.
>>
>> There's even another implementation of outbuffer in Phobos (because I 
>> thought outbuffer was generally very useful):
>>
>> https://dlang.org/phobos/std_outbuffer.html
>>
>> But here we run into our rule that dmd shouldn't rely on Phobos. 
>> Compromise is inevitable. Outbuffer isn't a spike we need to impale 
>> ourselves on. There are plenty of other encapsulation problems that 
>> could be improved, like target.d.
> 
> Outbuffer is a case of a data structure that is useful throughout the 
> compiler. So it is put in a package of the compiler that is OK to be 
> imported from other packages (and should avoid importing other 
> packages). I think the `dmd.root` package is exactly like such a 
> package  (compare with `ADT` in LLVM). From that standpoint, I don't see 
> why the `dmd.backend` package cannot import `dmd.root`. If `dmd.backend` 
> is to be used in different projects, then those should also use 
> `dmd.root` and that's where the dependency chain stops.

Thanks. I set out to write pretty much exactly that. To add to it:

A. High-level modules should not depend on low-level modules. Both 
should depend on abstractions (e.g., interfaces).
B. Abstractions should not depend on details. Details (concrete 
implementations) should depend on abstractions.

(Source: https://en.wikipedia.org/wiki/Dependency_inversion_principle)

Applied here:

A. The back-end should not depend on the front end. Both should depend 
on abstractions (e.g., interfaces) such as OutBuffer.
B. OutBuffer should not depend on memory-mapped minutia. Memory-mapped 
work should be done to serve OutBuffer.

> Better: if it is in Phobos, great, use that!
> If you need a certain data structure you know where to look: Phobos or 
> `dmd.root`. Is it not in there? Don't create a new structure elsewhere, 
> add it to `dmd.root` and import it.

I am sympathetic to the cause of not addding a large number of moving 
pieces to the compiler codebase. But yes the point stands.

Hopefully good versioning could help a lot with all that.

> In LDC, we use the C++ stdlib and we use Phobos, because why not? We are 
> programming in D after all, and it is the standard library that is 
> available in all bootstrapping compiler required versions. We do take 
> care not to rely on _latest_ Phobos, but on Phobos from the oldest 
> bootstrapping D compiler version supported up to the latest version. 
> Same for the C++ stdlib (C++20 is not ok, but C++14 is much encouraged).
> 
> For example: LDC uses MD5 hashing for its machine codegen cache. `import 
> std.digest.md; auto md5hash = md5Of(slice);`  Done.
> LLVM does the same, e.g. the project removed its own unique_ptr 
> implementation (`OwningPtr`), and now uses `std::unique_ptr`.

Cool stuff!

> My standpoint on the original topic of "make it easier to experiment 
> with the compiler": I disagree with making the code more stable. If 
> anything, we should be refactoring much more aggressively, changing 
> function names etc.

Doesn't aggressive refactoring require massive unittests?