Building the compiler in 2 seconds with `dmd -i`

[max haughton]

On Friday, 19 May 2023 at 09:51:39 UTC, Dennis wrote:
> In theory, dmd is really easy to build: it's just a collection 
> of D files in a single repository with no external 
> dependencies. Why have there always been such complex 
> makefiles/scripts to build it?
>
> Well, after replacing most of dmd's backend's C-style extern 
> function declarations with proper imports [1], there's a simple 
> way to build the compiler. Instead of needing to list every 
> source file, you can leverage the `-i` flag which automatically 
> compiles imports, and create a build script as short as this:
>
> ```
> echo -n "/etc" >> SYSCONFDIR.imp
> cd compiler/src
> dmd -i dmd/mars.d -of=../../dmd dmd/eh.d dmd/backend/dtype.d 
> -version=MARS -Jdmd/res -J../..
> ```
>
> On my linux pc, these 3 lines take 2.0 seconds to complete and 
> it uses max 680MB RAM.
>
> The official way to build a non-release compiler is:
>
> ```
> cd compiler/src
> rdmd build.d BUILD=debug
> ```
>
> build.d is 2400 lines, takes 2.3 seconds to complete a clean 
> build, and uses max 613MB RAM.
>
> The build script does separate compilation, multi-threading, 
> includes debug info (`-g`), and builds other things as well. 
> Still, my takeaways are:
>
> - The build script is overly complex
> - `dmd -i` is awesome
> - Without too much templates/CTFE, dmd compiles pretty fast 
> (150 KLOC/second in this case), reducing the need for 
> incremental/parallel compilation
>
> Eventually I hope to make just `dmd -i dmd/mars.d` work, I'll 
> see how close I can get to that.
>
> [1] 
> https://github.com/dlang/dmd/pulls?q=is%3Aclosed+is%3Apr+author%3Adkorpel+label%3ABackend+label%3ARefactoring+prototypes

I'm about 80% certain I have got mars.d working with -i before. I 
forget what I did specifically but it was basically the same 
stuff in the backend.

The reason why dmd -i works better than one might initially 
assume is that:

1. importing druntime is too slow.
2. Semantic analysis is not shared between parallel compiler 
invocations

Both of which result in a lower bound on the time taken 
regardless of parallelism

2.0 seconds is still way way way too slow IMO. Iteration should 
ideally be instant, tests running before you blink after saving 
the file. That's the benefit of incremental compilation, ideally 
you just have it watch the source code and reuse stuff that 
didn't change.