Reggae v0.0.5 super alpha: A build system in D

[Atila Neves via Digitalmars-d-announce]

On Friday, 3 April 2015 at 17:55:00 UTC, Dicebot wrote:
> On Friday, 3 April 2015 at 17:25:51 UTC, Ben Boeckel wrote:
>> On Fri, Apr 03, 2015 at 17:10:31 +0000, Dicebot via 
>> Digitalmars-d-announce wrote:
>>> On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:
>>> > . Separate compilation. One file changes, only one file 
>>> > gets rebuilt
>>> 
>>> This immediately has caught my eye as huge "no" in the 
>>> description. We must ban C style separate compilation, there 
>>> is simply no way to move forward otherwise. At the very least 
>>> not endorse it in any way.
>>
>> Why? Other than the -fversion=... stuff, what is really 
>> blocking this? I
>> personally find unity builds to not be worth it, but I don't 
>> see
>> anything blocking separate compilation for D if dependencies 
>> are set up
>> properly.
>>
>> --Ben
>
> There are 2 big problems with C-style separate compilation:
>
> 1)
>
> Complicates whole-program optimization possibilities. Old 
> school object files are simply not good enough to preserve 
> information necessary to produce optimized builds and we are 
> not in position to create own metadata + linker combo to 
> circumvent that. This also applies to attribute inference which 
> has become a really important development direction to handle 
> growing attribute hell.
>
> During last D Berlin Meetup we had an interesting conversation 
> on attribute inference topic with Martin Nowak and dropping 
> legacy C-style separate compilation seemed to be recognized as 
> unavoidable to implement anything decent in that domain.
>
> 2)
>
> Ironically, it is just very slow. Those who come from C world 
> got used to using separate compilation to speed up rebuilds but 
> it doesn't work that way in D. It may look better if you change 
> only 1 or 2 module but as amount of modified modules grows, 
> incremental rebuild quickly becomes _slower_ than full program 
> build with all files processed in one go. It can sometimes 
> result in order of magnitude slowdown (personal experience).
>
> Difference from C is that repeated imports are very cheap in D 
> (you don't copy-paste module content again and again like with 
> headers) but at the same time semantic analysis of imported 
> module is more expensive (because D semantics are more 
> complicated). When you do separate compilation you discard 
> already processed imports and repeat it again and again from 
> the very beginning for each new compiled file, accumulating 
> huge slowdown for application in total.
>
> To get best compilation speed in D you want to process as many 
> modules with shared imports at one time as possible. At the 
> same time for really big projects it becomes not feasible at 
> some point, especially if CTFE is heavily used and memory 
> consumption explodes. In that case best approach is partial 
> separate compilation - decoupling parts of a program as static 
> libraries and doing parallel compilation of each separate 
> library - but still compiling each library in one go. That 
> allows to get parallelization without doing the same costly 
> work again and again.

Interesting.

It's true that it's not always faster to compile each module 
separately, I already knew that. It seems to me, however, that 
when that's actually the case, the practical difference is 
negligible. Even if 10x slower, the linker will take longer 
anyway. Because it'll all still be under a second. That's been my 
experience anyway. i.e. It's either faster or it doesn't make 
much of a difference.

All I know is I've seen a definite improvement in my 
edit-compile-unittest cycle by compiling modules separately.

How would the decoupling happen? Is the user supposed to 
partition the binary into suitable static libraries? Or is the 
system supposed to be smart enough to figure that out?

Atila