Incremental compilation with DMD

[Tom S]

Walter Bright wrote:
> Tom S wrote:
>> Thus my suggestion of adding an option to DMD so it may emit template 
>> instances to all object files that use them. If anyone has alternative 
>> ideas, I'd be glad to hear them, because I'm running out of options.
> 
> Try compiling with -lib, which will put each template instance into its 
> own obj file.

Thanks for the suggestion. Unfortunately it's a no-go since -lib seems 
to have the same issue that compiling without -op does - if you have 
multiple modules with the same name (but different packages), one will 
overwrite the other in the lib. On the other hand, I was able to hack 
DMD a bit and use -multiobj since your suggestion gave me an idea :)

Basically, the approach would be to compile the project with -multiobj 
and move the generated objects to a local (per project) directory, 
renaming them so no conflicts arise.

The next step is to determine all public and comdat symbols in all of 
these object files - this might be done via a specialized program, 
however I've used Burton Radons' exelib to optimally run libunres.exe 
from DMC. The exports are saved to some sort of a database (a dumb 
structured file is ok).

The following is done on the initial build - so the next time we have 
some object files in a directory and a map of all their exported 
symbols. In an incremental step, we'll compile the modified modules, but 
don't move their object files immediately over to the special directory. 
We'll instead scan their public and comdat symbols and figure out which 
object files they replace from our already compiled set. For each symbol 
in the newly compiled objects, find which object in the original set 
defined it, then mark it. For all marked files, add them to a library ( 
I call it junk.lib ), then remove the source object. Finally, move the 
newly compiled objects to the special object directory.

The junk.lib will be used if the newly compiled object files missed any 
shared symbols that were in the old objects and that would be generated, 
had more modules be passed to the compiler. In other words, it contains 
symbols that the naive incremental compilation will lose.

When linking, all object files from the directory are passed explicitly 
to the compiler and symbols are pulled eagerly from them, however 
junk.lib will be queried only if a symbol cannot be found in the set of 
objects in the special directory.

I've put up a proof-of-concept implementation at 
http://h3.team0xf.com/increBuild.7z . It requires a slightly patched DMD 
(well, hacked actually), so it prints out the names of all objects it 
generates. Basically, uncomment the `printf("writing '%s'\n", fname);` 
in glue.c at line 133 and add `printf("writing '%s'\n", 
m->objfile->name->str);` after `m->genobjfile(global.params.multiobj);` 
in mars.c. I'm compiling the build tool with a recent (SVN-ish) version 
of Tango and DMD 1.047.

As for my own impressions of this idea, its biggest drawback probably is 
that the multitude of object files created via -multiobj strains the 
filesystem. Even when running on a ramdrive, my WinXP-based system took 
a good fraction of a second to move a few hundred object files to their 
destination directory. This can probably be improved on, as -multiobj 
seems to produce some empty object files (at least according to libunres 
and ddlinfo). It might also be possible to use specialized storage for 
object files by patching up dmd and hooking OPTLINK's calls to 
CreateFile. I'm not sure about Linux, but perhaps something based on 
FUSE might work. These last options are probably long shots, so I'm 
still quite curious how DMD might perform with outputting template 
instantiations into each object file that uses them.


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode