Compiler scalability. Question inspired by OOM errors seen by Crystal language

[via Digitalmars-d]

On Wednesday, 30 August 2017 at 16:39:32 UTC, H. S. Teoh wrote:
> On Wed, Aug 30, 2017 at 08:01:17AM +0000, via Digitalmars-d 
> wrote: [...]
>> D supports separate compilation by design. I.e. it doesn't 
>> require all the source files corresponding to all the object 
>> files being linked to produce the final executable, to be 
>> loaded in memory by the compiler.
>
> Yes, I think the general recommendation for very large 
> codebases is to compile one package at a time, i.e., one subdir 
> at a time.

Yep, that avoids parsing the same file over and over again (like 
would be the case in C++ with commonly used header files for each 
translation unit), yet provides enough granularity for 
parallelism and also somewhat limits the memory problem.

> Nevertheless, compiler memory usage remains an issue on 
> low-memory systems.  My old work PC simply cannot handle 
> running dmd without grinding to a halt because there's just not 
> enough RAM to go around (only 500MB total).  It's generally not 
> a problem for modern PCs with at least a few GB of memory.  
> Walter chose compilation speed over memory efficiency, so 
> that's just the way it is.  In theory, one could turn on GC in 
> the compiler so that it will work on low-memory systems, but

(Nods)

> I'm not sure if such a change will be accepted into dmd.
>
>
> T

We could always make that a compiler switch. I'm not sure if a 
substantial benefit could be gained during parsing and semantic 
analysis (e.g. consider staticIota - each instantiation needs to 
be kept in memory, since you never know when it will be needed 
again), but certainly once you convert the AST to the backend IR, 
you could immediately drop all memory allocated by the frontend.