[Dlang-internal] multi threading in dmd

[Jacob Carlborg]

On 2019-10-11 14:49, Robert Schadek wrote:
> Compiling is IMHO is getting painfully slow with growing projects.
> One thing I'm working on is up to 30+ seconds, for 20k lines of somewhat 
> heavy code.
> But lets not argue whether or not or not I'm doing it wrong, for the 
> sake of
> the arguments lets assume compiling is slow.
> 
> One thing I see is that dub passes many files at once to dmd.
> And dmd runs one thread on that input.
> 
> I think there is some opportunity to start multiple threads to do at 
> least some of
> the work in parallel.
> 
> 1. Has anybody done any work on doing work in dmd with threads?
> 
> 2. Am I correct that in theory dmd should be able to lex all passed 
> files in
> parallel (given enough cpu cores).
> 
> 3. Is it correct that currently one token is created at a time on 
> request by the
> parser.
> 
> 4. This would currently require the classes Identifier and StringTable 
> be made
> thread safe.
> 
> 5. AsyncRead in mars.d is dead code?
> 
> 6. Is there any way to test all the different version statements and 
> static if's
> used for the same purpose?
> 
> 7. Is there a change to parse all the initially given files in parallel?
> 
> 8. Any other ideas on how to do threading in dmd?

In general, there are quite a lot of globals in DMD. There are five 
`__gshared` variables in the lexer alone. Then more will be referenced 
from the lexer and parser. There's also the issue of reporting 
diagnostics. That might need to be synchronized otherwise some parts of 
an error message from one file might be printed and then some other 
parts from another file.

Most of the time is spent doing semantic analysis. That will be even 
harder to do with multiple threads.

-- 
/Jacob Carlborg