Reggae v0.0.5 super alpha: A build system in D

[Dicebot via Digitalmars-d-announce]

On Friday, 3 April 2015 at 17:25:51 UTC, Ben Boeckel wrote:
> On Fri, Apr 03, 2015 at 17:10:31 +0000, Dicebot via 
> Digitalmars-d-announce wrote:
>> On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:
>> > . Separate compilation. One file changes, only one file gets 
>> > rebuilt
>> 
>> This immediately has caught my eye as huge "no" in the 
>> description. We must ban C style separate compilation, there 
>> is simply no way to move forward otherwise. At the very least 
>> not endorse it in any way.
>
> Why? Other than the -fversion=... stuff, what is really 
> blocking this? I
> personally find unity builds to not be worth it, but I don't see
> anything blocking separate compilation for D if dependencies 
> are set up
> properly.
>
> --Ben

There are 2 big problems with C-style separate compilation:

1)

Complicates whole-program optimization possibilities. Old school 
object files are simply not good enough to preserve information 
necessary to produce optimized builds and we are not in position 
to create own metadata + linker combo to circumvent that. This 
also applies to attribute inference which has become a really 
important development direction to handle growing attribute hell.

During last D Berlin Meetup we had an interesting conversation on 
attribute inference topic with Martin Nowak and dropping legacy 
C-style separate compilation seemed to be recognized as 
unavoidable to implement anything decent in that domain.

2)

Ironically, it is just very slow. Those who come from C world got 
used to using separate compilation to speed up rebuilds but it 
doesn't work that way in D. It may look better if you change only 
1 or 2 module but as amount of modified modules grows, 
incremental rebuild quickly becomes _slower_ than full program 
build with all files processed in one go. It can sometimes result 
in order of magnitude slowdown (personal experience).

Difference from C is that repeated imports are very cheap in D 
(you don't copy-paste module content again and again like with 
headers) but at the same time semantic analysis of imported 
module is more expensive (because D semantics are more 
complicated). When you do separate compilation you discard 
already processed imports and repeat it again and again from the 
very beginning for each new compiled file, accumulating huge 
slowdown for application in total.

To get best compilation speed in D you want to process as many 
modules with shared imports at one time as possible. At the same 
time for really big projects it becomes not feasible at some 
point, especially if CTFE is heavily used and memory consumption 
explodes. In that case best approach is partial separate 
compilation - decoupling parts of a program as static libraries 
and doing parallel compilation of each separate library - but 
still compiling each library in one go. That allows to get 
parallelization without doing the same costly work again and 
again.