Honey, I shrunk the build times

[Andrei Alexandrescu via Digitalmars-d]

On 6/6/15 5:45 PM, Jonathan M Davis wrote:
> On Saturday, 6 June 2015 at 21:42:47 UTC, Andrei Alexandrescu wrote:
>> https://github.com/D-Programming-Language/phobos/pull/3379
>>
>> Punchline: major reduction of both total run time and memory consumed.
>
> Reading makefiles always gives me a headache. For those of us who can't
> just glance through the changes and quickly decipher them, what did you
> actually do?

Thanks for asking. The situation before went like this: to build 
libphobos2.a, the command would go like this (simplified to just a few 
files and flags:

dmd -oflibphobos2.a std/datetime.d std/conv.d std/algorithm/comparison.d 
std/algorithm/iteration.d

So all modules would go together in one command to build the library. 
With the package-at-a-time approach, we build one directory at a time 
like this:

dmd -oflibphobos2_std.a std/datetime.d std/conv.d
dmd -oflibphobos2_std_algorithm.a std/algorithm/comparison.d 
std/algorithm/iteration.d

So now we have two libraries that need to be combined together, which is 
easy:

dmd -oflibphobos2_std.a libphobos2_std.a libphobos2_std_algorithm.a

and voila, the library is built.

This is strictly speaking more work:

* Everything in Phobos imports everything else, so effectively we're 
parsing the entire Phobos twice as much

* There are temporary files being created

* There's an extra final step - files that have just been written need 
to be read again

However, the key advantage here is that the first two steps can be 
performed in parallel, and that turns out to be key. Time and again I 
see this: parallel processing almost always ends up doing more work - 
some of which is wasteful, but in the end it wins. It's counterintuitive 
sometimes.

This is key to scalability, too. Now, the baseline numbers were without 
std.experimental.allocator. Recall the baseline time on my laptop was 
4.93s. I added allocator, boom, 5.08s - sensible degradation. However, 
after I merged the per-package builder I got the same 4.01 seconds.


Andrei