Incremental compilation with DMD

[Robert Jacques]

On Fri, 11 Sep 2009 07:47:11 -0400, Tom S  
<h3r3tic at remove.mat.uni.torun.pl> wrote:
> Short story: DMD probably needs an option to output template instances  
> to all object files that need them.
>
> Long story:
>
> I've been trying to make incremental compilation in xfBuild reliable,  
> but it turns out that it's really tricky with DMD. Consider the  
> following example:
>
> * module A instantiates template T from module C
> * module B instantiates the same template T from module C (with the same  
> arguments)
> * compile all modules at the same time in the order: A, B, C
> * now A.obj contains the instantiation of T
> * remove the instantiation from the A module
> * perform an incremental compilation - 'A' was changed, so only it has  
> to be recompiled
> * linking of A.obj, B.obj and C.obj fails because no module has the  
> instantiation of T for B.obj
>
> What happens is that the optimization in DMD to only emit templates to  
> the first module that needs it creates implicit inter-module  
> dependencies. I've tried tracking them by modifying DMD, but still  
> wouldn't find them all - it seems that one would have to dig deep in the  
> codegen, my attempts at hacking the frontend (mostly template.c) weren't  
> enough.
>
> Yet, I still managed to get some of these implicit dependencies figured  
> and attempted using this extra info in xfBuild when deciding what to  
> compile incrementally. I've tossed it on a project of mine with > 350  
> modules and no circular imports. The result was that even a trivial  
> change caused most of the project to be pulled into compilation.
>
> When doing regular incremental compilation, all modules that import the  
> modified ones must be recompiled as well. And all modules that import  
> these, and so on, up to the root of the project. This is because the  
> incremental build tool must assume that the modules that import module  
> 'A' could have code of the form 'static if (A.something) { ... } else {  
> ... }' or another form of it. As far as I know, it's not trivial to  
> detect whether this is really the case or whether the change is isolated  
> to 'A'.
>
> When trying to cope with the implicit dependencies caused by template  
> instantiations and references, one also has to recompile all modules  
> that contain template references to a module/object file which gets the  
> instance. In the first example, it would mean recompiling module 'B'  
> whenever 'A' changes. The graph of dependencies here doesn't depend very  
> much on the structure of imports in a project, but rather in the order  
> that DMD decides to run semantic() on template instances.
>
> Add up these two conservative mechanisms and it turns out that tweaking  
> a simple function causes half of your project to be rebuilt. This is not  
> acceptable. Even if it was feasible - getting these implicit  
> dependencies is probably a matter of either hacking the backend or  
> dumping object files and matching unresolved symbols with comdats.  
> Neither would be very fast or portable.
>
> Compiling modules one-at-a-time is not a solution because it's too slow.
>
> Thus my suggestion of adding an option to DMD so it may emit template  
> instances to all object files that use them. If anyone has alternative  
> ideas, I'd be glad to hear them, because I'm running out of options. The  
> approach I'm currently using in an experimental version of xfBuild is:
>
> * get a fixed order of modules to be compiled determined by the order  
> DMD calls semantic() on them with the root modules at the end
> * when a module is modified, additionally recompile all modules that  
> occur after it in the list
>
> This quite obviously ends up compiling way too many modules, but seems  
> to work reliably (except when OPTLINK decides to crash) without  
> requiring full rebuilds all the time. Still, I fear there might be  
> corner cases where it will fail as well. DMD sometimes places  
> initializers in weird places, e.g.:
>
> .objs\xf-nucleus-model-ILinkedKernel.obj(xf-nucleus-model-ILinkedKernel)
>   Error 42: Symbol Undefined  
> _D61TypeInfo_S2xf7nucleus9particles13BasicParticle13BasicParticle6__initZ
>
> The two modules (xf.nucleus.model.ILinkedKernel and  
> xf.nucleus.particles.BasicParticle) are unrelated. This error occured  
> once, somewhere deep into an automated attempt to break the experimental  
> xfBuild by touching random modules and performing incremental builds.
>
>

On the other hand, one-at-a-time builds can be done in parallel if you  
have multi-cores. Of course, still not a net win on my system, so vote++