Compilation is taking a ton of memory

[Steven Schveighoffer]

On 6/27/18 12:00 PM, Mario Silva wrote:
> Hello,
> 
> Our code base has been growing steadily and it's currently at a point 
> where my 16GB machine just freezes when we're compiling our code. This 
> happens because DMD just consumes all my memory for a while.

That is the path of D's compiler toolchain. The front end uses a simple 
"bump the pointer" memory allocation scheme, and NEVER frees memory. 
This is reasonable for many things in the compiler (e.g. you never want 
to forget about symbols you built), but the worst is if you have a lot 
of CTFE, as that consumes a lot of memory that could easily be reclaimed 
after it's done running.

> Also, it's taking a long time to compile it. Less than an year ago our 
> project was taking around 17 seconds to compile - no libs requiring 
> compilation - and maybe around 50 seconds for full compilation, and it 
> now takes around 50 seconds for an incremental compilations and around 
> 1.5 minutes for a full one.

Yikes, I experience stuff like this, and I'm using vibe.d as well. But I 
haven't gotten big enough to make it horrible. I'm compiling my vibe.d 
project on a VM Linux image with only 2GB of RAM.

It's definitely a problem for template-heavy or CTFE heavy builds.

> 
> For you guys to have an idea of the size of our project, we have 21151 
> lines of code and then 50933 in our libs. This is just our code without 
> counting dependencies fetched by dub like vibe.d for example.

The libraries should build OK. It's really the templates or diet 
templates that take the longest. And that's not a large project in 
comparison to other projects.

> Are there any plans to work on compiler performance in terms of memory 
> consumption and compilation time?

This has been a thorn in many sides for a long time. I remember 
Weka.io's Liran talking about how they required an INSANE amount of 
time/memory to build their system in dconf 2015 maybe? But things have 
gotten a bit better since then. I think at some point there will be a 
reckoning where it has to be fixed. Fast compile-time is cool, but 
inifinte compile time (i.e. it never finishes because the OOM killer 
destroys your process) is not acceptable.

> Any tips on how to code in a way that minimizes both compilation times 
> and memory consumption when compiling?

The default for dub is to build "separately", you can try different 
build modes --build-mode=allAtOnce or --build-mode=singleFile. There is 
very scarce documentation as to what these actually do.

I don't know if these would help at all, but worth a try. Note that in 
some respects, memory usage and compilation times are related. If you 
want to keep memory usage low, you may have to forget some things, and 
then re-figure them out later, which slows you down. But if you consume 
too much memory, your system thrashes and then you experience a slowdown 
that way.

-Steve