D compilation is too slow and I am forking the compiler

[welkam]

On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright 
wrote:
> Wouldn't it be awesome to have the lexing/parsing of the 
> imports all done in parallel?

 From my testing lexing/parsing takes small amount of build time 
so running it in parallel might be small gain. We should consider 
running in parallel more heavy hitting features like CTFE and 
templates.

Since we are in wish land here is my wishes. Currently D reads 
the all files that are passed in command line before starting 
lexing/parsing, but in principle we could start lexing/parsing 
after first file is read. In fact we could start after first 
file`s first line is read. Out of all operation before semantic 
pass, reading from hard disk should be the slowest so it might be 
possible to decode utf-8, lex and parse at the speed of reading 
from hard disk. If we run these steps in different thread on the 
same core with SMT we could better use core`s resources. Reading 
file with kernel, decoding UTF-8 with vector instructions and 
lexing/parsing with scalar operations while all communication is 
done trough L1 and L2 cache.

I thought about using memory mapped files to unblock file reading 
as a first step but lack of good documentation about mmf and lack 
of thorough understanding of front end made me postpone this 
modification. Its a change with little benefit.

> The main difficulty in getting that to work is dealing with the 
> shared string table.

At begging of parsing a thread could get a read-only shared slice 
of string table. All strings not in table are put in local string 
table. After parsing tables are merged and shared slice is 
updated so new thread could start with bigger table. this assumes 
that table is not sorted