Is run.d going to be expand for runtime and the phobos library?

[Seb]

On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
wrote:
> The original Andrei, just like today's Andrei, has an 
> appreciation for good engineering. I didn't feel the need to 
> add to provide detail because (a) most regulars in this forum 
> already knew what I was going to say, and (b) nobody save for a 
> few would share my opinion.
>
> But, I'll bite again, again to regret it.

I'll reply with similar regrets, because I agree that this 
discussion isn't helping anyone.
tl;dr: I agree with you that build.d isn't the ideal engineering 
piece that D was supposed to enable, but it did solve a few real 
problems and using D without dependency was really the only 
option to move forward:

In Detail:

1) DigitalMars Make was a massive real problem (and still is for 
Druntime+Phobos)
2) Duplication into many similar Makefiles with mismatching 
definitions (posix.mak, win32.mak, win64.mak, osmodel.mak, ...)
3) There were many things we couldn't model properly in Make 
(e.g. version detection shell script, documentation build, ...)
4) Especially the Windows Makefiles were missing a lot of target 
and options (e.g. no LTO/PGO builds, no library builds, ...)

https://github.com/dlang/dmd/blob/v2.080.0/src/posix.mak
https://github.com/dlang/dmd/blob/v2.080.0/src/win32.mak
https://github.com/dlang/dmd/blob/v2.080.0/src/win64.mak

5) You and Walter repeatedly struck down all attempts to use any 
other build tool (like e.g. reggae for Phobos - 
https://github.com/dlang/phobos/pull/4194)
6) The Windows Makefiles looked liked this:

---
$G/nteh.obj : $C\rtlsym.h $C\nteh.c
	$(CC) -c -o$@ $(MFLAGS) $C\nteh

$G/os.obj : $C\os.c
	$(CC) -c -o$@ $(MFLAGS) $C\os

$G/out.obj : $C\out.c
	$(CC) -c -o$@ $(MFLAGS) $C\out

$G/outbuf.obj : $C\outbuf.h $C\outbuf.c
	$(CC) -c -o$@ $(MFLAGS) $C\outbuf

$G/pdata.obj : $C\pdata.c
	$(CC) -c -o$@ $(MFLAGS) $C\pdata

$G/ph2.obj : $C\ph2.c
	$(CC) -c -o$@ $(MFLAGS) $C\ph2
---

7) Also, complexity in general increased over time as C++ header 
generation or CXX compatibility tests got added. There are a 
couple of other new features that are quite helpful for CIs (like 
e.g. the very detailed failure message on build failures).

In build.d there definitely are a few relicts from the Makefiles 
times as it was a lot easier to do a step-by-step migration of 
the targets than migrating everything in one bulk, so I'm not 
denying that the build script source code could be refactored and 
improved, but so could be almost any code in the D ecosystem.

Lastly, if you really want to know why build.d grew from 250 
lines initially to its current size, the answer is rather simple. 
It's because of many, many shortcomings and bugs in Druntime and 
Phobos.
As a prime example I encourage you to read through is:

https://github.com/dlang/dmd/pull/10611 (GitHub hides most of the 
discussion by default)

> * Then, we have main() dealing with a lot of stuff. (Why?)

Not sure, what's a "a lot of stuff" for you, but that's the main 
method as of now:

---
int main(string[] args)
{
     try
     {
         runMain(args);
         return 0;
     }
     catch (BuildException e)
     {
         // Ensure paths are relative to the root directory
         // s.t. error messages are clickable in most IDE's
         writeln(e.msg.replace(buildPath("dmd", ""), 
buildPath("src", "dmd", "")));
         return 1;
     }
}
---

To be fair, runMain does more things (gettings arguments + 
starting thread pool) and of course, the showHelp() could have 
easily been moved outside, but I really don't see how this is a 
critical problem for D?
Especially considering how big main() is at "your" rdmd:

https://github.com/dlang/tools/blob/master/rdmd.d#L62

> * The rules using makeRule are a confusing mix of code and data 
> starting at line 249, after the reader has browsed through a 
> bunch of implementation details.

1) Those are all rules. The implementation details are the 
rules...
2) The "reader" has a nice overview of all rules and targets at 
the beginning (before implementation)
3)) A typical "reader" is much more likely to execute ./build.d 
-h (or --help) than reading the source

> An interesting side question  would be what language features
> would make such things easier to write; e.g. a mixin string
> in conjunction with a DSL that implements a small subset of 
> make may be nicer.

Maybe a standard library with isn't riddled with hundreds of bugs 
and old/outdated/unusable code?

> But also becomes ironic - we started by going away from make.)

It's not ironic. WE CAN'T USE MAKE ON WINDOWS.

> * Hundreds of lines of rules follow.

Most of these rules are to support ugly things like e.g. the 
complicated ddoc or to workaround shortcomings in the standard 
library (e.g. no safe cross-platform scheduler).

> Did build.d attain its objective?

Absolutely. One big example is that people can now build DMD on 
Windows (this was _very_ hard/near to impossible before).

> * The source files are hardcoded somewhere starting at line 
> 1160. I wish I was kidding.

build.d used globbing initially. It was removed, because Walter 
explicitly requested hard-coding.
I can't find the PR with the request at the moment, but here's 
one of the PRs which removed the globbing in favor of hard-coding:

https://github.com/dlang/dmd/pull/10212

> * A lot more support code follows, which is fine for a 
> single-file build tool. I wish this were a two-files affair a 
> la build.d and buildinfo.d, or if single file have an obvious 
> separation:

While we could have done a separation for rdmd which you struck 
done a while ago [1] which btw lead to the only two people 
interested in contributing to rdmd loosing their interest.

Now, to answer your question fully I have to add that one of 
those two people (Jonathan Marler) has since then decided to 
create a better rdmd: rund [2]. Rund fixes one of the main 
shortcomings of rdmd (the need to compile a file twice with dmd 
and thus having an almost doubled run time) and adds a couple of 
other handy features like being able to specify execution flags, 
e.g.

---
#!/usr/bin/env rund
//!importPath <path>
---

If rdmd would have such a feature, we could split build.d into 
multiple files. Unfortunately, it doesn't and thus it must remain 
a single file as `./build.d` needs to work out of the box.

[1] https://github.com/dlang/tools/pull/344
[2] https://github.com/dragon-lang/rund

> or if single file have an obvious separation:

The separation is already there [1] , but I guess one could make 
it more obious.

[1] https://github.com/dlang/dmd/blob/master/src/build.d#L768

> I've often said gmake should be used for Windows to build dmd. 
> Now we require dmd to build dmd...

Yes, how else is one going to build D code since 2.067?

> Of course that ruined the carefully constructed AUTO_BOOTSTRAP 
> option that allows building dmd on a fresh system.

Nope. In fact, AUTO_BOOTSTRAP is still being used by some of our 
own CIs, e.g. Buildkite still does

```
make -f posix.mak AUTO_BOOTSTRAP=1
```

Some other do this to get an active D compiler in your 
environment:

```
source $(curl https://dlang.org/install.sh | bash -s install -a)
```

> Not when I tried it: 
> https://forum.dlang.org/post/r5e8uj$i1b$1@digitalmars.com

To be fair that's not a fault of build.d. You aren't able to 
compile anything with an outdated compiler on OSX. For details 
see some of the responses to your thread:

https://forum.dlang.org/post/rimiuawxnzobcgfuwxez@forum.dlang.org
https://forum.dlang.org/post/zvsjvhvgprhgacybvctb@forum.dlang.org