Compilation is taking a ton of memory

[H. S. Teoh]

On Wed, Jun 27, 2018 at 04:00:37PM +0000, Mario Silva via Digitalmars-d wrote:
> Hello,
> 
> Our code base has been growing steadily and it's currently at a point
> where my 16GB machine just freezes when we're compiling our code. This
> happens because DMD just consumes all my memory for a while.

How are you compiling the code? Dub? Makefile? Are you compiling
everything all at once?

DMD is known to prefer compilation speed over memory efficiency, a
choice I personally don't quite agree with, but it is what it is.  That
means past a certain size, you're going to have to split up your
project into multiple separate compilations.  The usual convention is to
compile packages together, e.g., if you have the source tree structure:

	main/main.d
	main/common.d
	pkg1/package.d
	pkg1/mod1.d
	pkg1/mod2.d
	pkg2/package.d
	pkg2/mod1.d
	pkg2/mod2.d

then you'd compile main/* first, say into main.a (or main.lib), and
compile pkg1/* into pkg1.a, and compile pkg2/* into pkg2.a, then finally
link them all together with a final invocation of the compiler. This
should reduce the total amount of memory used by any single invocation
of dmd, which should keep memory usage sane.  It may also reduce
compilation time if only one package needs to be recompiled (though
template-heavy code may not save that much, depending on what you do
with it).


[..]
> Are there any plans to work on compiler performance in terms of memory
> consumption and compilation time?

Not that I know of, though it'd be really nice if somebody worked on
that.  Given Walter's past choice of compilation speed over memory
efficiency, I'm not sure how far such an effort would get.  But I'd vote
for it.

There *is* the option of just turning on the GC, now that dmd is
bootstrapping, but the last time I tried there were bugs that caused
crashes, so it may not (yet) be a viable option.


> Any tips on how to code in a way that minimizes both compilation times
> and memory consumption when compiling?
[...]

In general: (1) avoid template bloat, and (2) use separate compilation
in strategic ways.

(1) implies:

- Avoid using (or reduce usage of) libraries / modules that are heavy on
  templates, except where absolutely necessary.  If you're using Phobos,
  std.format comes to mind (though in a sufficiently large project I'd
  expect it doesn't play that big of a role, but the principle holds).
  If you're using vibe.d, Diet templates can be a source of heavy
  template usage.

  More generally, avoid recursive templates, or templates instantiation
  chains that nest too deeply. If not done carefully, you may end up
  with a quadratic or an exponential number of template instantiations,
  which will quickly eat up all your memory and CPU cycles (in addition
  to generating bloated executables).

- Avoid doing too much in CTFE, because the current CTFE engine is slow
  (and also very memory-inefficient).  We're all waiting for newCTFE to
  land, but from what I can tell that's still a ways away. (Note: not to
  be confused with other compile-time features that are not directly
  related to CTFE. Using static if's and the like won't add much
  overhead unless you're using recursive templates, in which case see
  point #1.)

If template-heavy or CTFE-heavy code is essential to your project,
consider splitting them into their own self-contained (sub)modules, and
separately compile them, so that recompilations that don't touch those
parts of the code will be reasonably fast.

In one of my vibe.d projects, I basically separated out the "business
logic" of the program into a module outside the code that directly
interfaces with vibe.d (Diet templates being the primary target). This
has sped up recompilations when only the "business logic" parts of the
code needs to change.  Furthermore, because Diet templates tend to get
very heavy, I also split those up into different compilation groups
(e.g., new user / account management pages, error/miscellaneous pages,
main pages) so that if I'm working on Diet templates in, say, the new
user creation pages, I only have to recompile the Diet templates related
to account management, not every other Diet template in the project.
This also saves a lot of time waiting for compilation.


T

-- 
It is impossible to make anything foolproof because fools are so ingenious. -- Sammy