Multithreaded I/O in the DMD compiler (DDJ article by Walter)

[Robert Fraser]

Brad Roberts wrote:
> BCS wrote:
>> Hello Daniel,
>>
>>> http://dobbscodetalk.com/index.php?option=com_myblog&show=Multithreade
>>> d-I-O.html&Itemid=29
>>>
>>> -- Daniel
>>>
>>> P.S.  "It really seems barbaric in our digital age that we all have
>>> motors and levers and gears and spinning things in our boxes."  Those
>>> damned spinning things!
>>>
>> Some sort of fixed size thread pool might work better than
>> thread-per-file after the files are loaded as the overhead of threads
>> isn't going to do you any good if you have many more threads than you
>> have cores
>>
> 
> Careful.. that rule of thumb applies to compute bound use of threads.
> 
> In this example, it's parallelization to achieve file io activity
> overlapping with compute activity.  If we were talking about physical
> disks, I'd say that one thread dedicated to "while(filetoread)
> open/read/close" would be sufficient.  Two would just add competing
> seeks.  But the picture likely changes once the files are in page cache.
> And SSD's are closer to memory than spindles, so it's unclear what the
> 'right' threading model there is too.  Regardless, with the model of
> most compilers today, as long as the data is ready by the time the main
> thread is ready for it, it's good enough.
> 
> It's fun to envision a really seriously multi-threaded compiler, but,
> like for any app, it requires a lot more care.. and dmd would need some
> cleanup to not use global state, but not a whole lot from the quick
> glances I've made at the source.  The devil, as always, is in the
> details.  There's enough to do and enough bugs to fix already that I
> can't see it as worth the investment right now, though.
> 
> Later,
> Brad

Compilers seem like they'd be easier to parallelize than other 
applications... but I think the last version of DSSS launches as many 
DMD processes as there are cores anyway.