Purity, memoization and parallelization of dmd

[Petar]

On Monday, 20 July 2020 at 10:39:08 UTC, Atila Neves wrote:
> On Thursday, 16 July 2020 at 18:21:11 UTC, Per Nordlöw wrote:
>> What's the status/progress on making dmd (completely) pure?
>>
>> Is this task on somebody's agenda? If so, are there any big 
>> obstacles that currently has no clear solution or is just a 
>> very large pile of small ones?
>>
>> And, in the long run, will a pure compiler (finally) enable 
>> caching/memoization of, for instance, template 
>> instantiations/ctfe-evaluations and, perhaps further into 
>> future, parallelization of the compiler?
>
> I don't think making the compiler parallel is particularly 
> important since that should be handled at the build-system 
> level (and if you use reggae, already is).

Build system level parallelism usually implies separate 
compilation (especially in the C++ world), however, if you're 
building at package-level granularity parallelism could be quite 
useful actually.
If you have a package, where many of its modules import each 
other, module-level separate compilation can be quite 
inefficient. For example, if a module has an immutable variable, 
the result of an expensive CTFE calculation, with separate 
compilation you would end up repeating the calculation every time 
this module is imported. With package-level compilation, it would 
be calculated only once.

I'd also say that build system-level caching is leaves a lot to 
be desired. At work we use various languages and frameworks where 
the compiler runs as a daemon process, listening for changes and 
then only recompiles parts of the program that changed. How big 
are the parts depends on the compiler implementation - it could 
be a file granularity, function granularity, or even a 
statement/expression granularity.

C#
https://github.com/dotnet/roslyn/wiki/EnC-Supported-Edits
https://joshvarty.com/2016/04/18/edit-and-continue-part-1-introduction/
https://joshvarty.com/2016/04/21/edit-and-continue-part-2-roslyn/

TS:
https://github.com/microsoft/TypeScript/wiki/Using-the-Compiler-API#incremental-build-support-using-the-language-services
https://github.com/microsoft/TypeScript/wiki/Using-the-Language-Service-API

Dart / Flutter:
https://flutter.dev/docs/development/tools/hot-reload
https://github.com/dart-lang/sdk/wiki/Hot-reload

Rust:
https://github.com/rust-lang/rfcs/blob/master/text/1298-incremental-compilation.md
https://blog.rust-lang.org/2016/09/08/incremental.html
https://internals.rust-lang.org/t/incremental-compilation-beta/4721
https://github.com/rust-lang/rust/issues/57968
https://blog.mozilla.org/nnethercote/2020/04/24/how-to-speed-up-the-rust-compiler-in-2020/