Practical parallelization of D compilation

[H. S. Teoh]

On Wed, Jan 08, 2020 at 04:40:02AM +0000, Guillaume Lathoud via Digitalmars-d-learn wrote:
> Hello,
> 
> One of my D applications grew from a simple main and a few source
> files to more than 200 files. Although I minimized usage of
> templating and CTFE, the compiling time is now about a minute.
> 
> I did not find any solution to take advantage of having multiple
> cores during compilation, lest I would write a makefile, or split
> the code into multiple packages and use a package manager.
> 
> (If I missed such a possibility, feel free to write it here.)
[...]

Generally, the recommendation is to separately compile each package.
E.g., if you have a source tree of the form:

	src/
	src/main.d
	src/pkg1/mod1.d
	src/pkg1/mod2.d
	src/pkg2/mod3.d
	src/pkg2/mod4.d

then you'd have 3 separate compilations:

	dmd -ofpkg1.o src/pkg1/mod1.d src/pkg1/mod2.d
	dmd -ofpkg2.o src/pkg2/mod3.d src/pkg2/mod4.d
	dmd -ofmyprogram src/main.d pkg1.o pkg2.o

The first two can be done in parallel, since they are independent of
each other.

The reason per-package granularity is suggested is because the
accumulated overhead of separately compiling every file makes it
generally not worth the effort.  D compiles fast enough that per-package
compilation is still reasonably fast, but you no longer incur as much
overhead from separately compiling every file, yet you still retain the
advantage of not recompiling the entire program after every change.

(Of course, the above example is greatly simplified; generally you'd
have about 10 or more files per package, and many more packages, so the
savings can be quite significant.)


T

-- 
Obviously, some things aren't very obvious.