Compilation strategy

[RenatoUtsch]

On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh wrote:
> On Sat, Dec 15, 2012 at 06:31:17PM +0100, RenatoUtsch wrote:
>> On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander
>> wrote:
>> >On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder
>> >wrote:
>> >>A quick straw poll.  Do people prefer to have all sources 
>> >>compiled
>> >>in a single compiler call, or (more like C++) separate 
>> >>compilation
>> >>of each object followed by a link call.
>> >
>> >Single compiler call is easier for small projects, but I worry
>> >about compile times for larger projects...
>> 
>> Yes, I'm writing a build system for D (that will be pretty damn
>> good, I think, it has some interesting new concepts), and 
>> compiling
>> each source separately to an object, and then linking 
>> everything
>> will allow easily to make the build parallel, dividing the 
>> sources
>> to compile in various threads. Or the compiler already does 
>> that if
>> I pass all source files in one call?
> [...]
>
> I find that the current front-end (common to dmd, gdc, ldc) 
> tends to
> work better when passed multiple source files at once. It tends 
> to be
> faster, presumably because it only has to parse 
> commonly-imported files
> once, and also produces smaller object/executable sizes -- 
> maybe due to
> fewer duplicated template instantiations? I'm not sure of the 
> exact
> reasons, but this behaviour appears consistent throughout dmd 
> and gdc,
> and I presume also ldc (I didn't test that). So based on this, 
> I'd lean
> toward compiling multiple files at once.

Yeah, I did read about this somewhee.

> However, in very large project, clearly this won't work very 
> well. If it
> takes half an hour to build the entire system, it makes the 
> code -
> compile - test cycle very slow, which reduces programmer 
> productivity.
>
> So perhaps one possible middle ground would be to link packages
> separately, but compile all the sources within a single package 
> at once.
> Presumably, if the project is properly organized, recompiling a 
> single
> package won't take too long, and has the perk of optimizing for 
> size
> within packages. This will probably also map to SCons easily, 
> since
> SCons builds per-directory.
>
>
> T

Well, the idea is good. Small projects usually don't have much 
packages, so there will be just a few compiler calls. And 
compiling files concurrently will only have a meaningful efect if 
the project is large, and a large project will have a lot of 
packages.

Maybe adding an option to choose between compiling all sources at 
once, per package, or per source. For example, in development and 
debug builds the compilation is per file or package, but in 
release builds all sources are compiled at once, or various 
packages at once.

This way release builds will take advantage of this behavior that 
the frontend has, but developers won't have productivity issues. 
And, of couse, the behaviour will not be fixed, the devs that are 
using the build system will choose that.