On the performance of building D programs

[Nick Sabalausky]

On Fri, 05 Apr 2013 00:49:08 +0200
"Vladimir Panteleev" <vladimir at thecybershadow.net> wrote:

> Recently I studied the performance of building a vibe.d example:
> https://github.com/rejectedsoftware/vibe.d/issues/208
> 
> I wrote a few tools in the process, perhaps someone might find 
> them useful as well.
> 
> However, I'd also like to discuss a related matter:
> 
> I noticed that compiling D programs in the usual manner (rdmd) is 
> as much as 40% slower than it can be.
> 
> This is because before rdmd knows a full list of modules to be 
> built, it must run dmd with -v -o-, and read its verbose output. 
> Then, it feeds that output back to the compiler again, and passes 
> all modules on the command line of the second run.
> 
> The problem with this approach is that DMD needs to parse, lex, 
> run CTFE, instantiate templates, etc. etc. - everything except 
> actual code generation / optimization / linking - twice. And code 
> generation can actually be a small part of the total compilation 
> time.
> 
> D code already compiles pretty quickly, but here's an opportunity 
> to nearly halve that time (for some cases) - by moving some of 
> rdmd's basic functionality into the compiler. DMD already knows 
> which modules are used in the program, so it just needs two new 
> options: one to enabled this behavior (say, -r for recursive 
> compilation), and one to specify an exclusion list, to indicate 
> which modules are already compiled and will be found in a library 
> (e.g. -rx). The default -rx settings can be placed in 
> sc.ini/dmd.conf. I think we should seriously consider it.
> 
> Another appealing thing about the idea is that the compiler has 
> access to information that would allow it to recompile programs 
> more efficiently in the future. For example, it would be possible 
> to get a hash of a module's public interface, so that a change in 
> one function's code would not trigger a recompile of all modules 
> that import it (assuming no CTFE).

About a year or two ago, Andrei proposed a system that addressed that
issue together with a substitute for a package manager. I was against
it for various reasons, and for the psudo-package-manager part of it I
still am. But I've since become more open to the "Make RDMD only call
DMD once" part. *That* part *might* not be a bad idea, although I can't
remember quite how it worked. Something about passing DMD a cmd line
tool to invoke whenever it finds a newly imported module? Although if
it was like that, that it could very well slow things down on Windows,
as Windows is notoriously slow at launching processes.

Or just fold RDMD functionality into DMD itself, like you
said...which would probably be easier anyway.