problem with parallel foreach

[Gerald Jansen via Digitalmars-d-learn]

On Wednesday, 13 May 2015 at 13:40:33 UTC, John Colvin wrote:
> On Wednesday, 13 May 2015 at 11:33:55 UTC, John Colvin wrote:
>> On Tuesday, 12 May 2015 at 18:14:56 UTC, Gerald Jansen wrote:
>>> On Tuesday, 12 May 2015 at 16:35:23 UTC, Rikki Cattermole 
>>> wrote:
>>>> On 13/05/2015 4:20 a.m., Gerald Jansen wrote:
>>>>> At the risk of great embarassment ... here's my program:
>>>>> http://dekoppel.eu/tmp/pedupg.d
>>>>
>>>> Would it be possible to give us some example data?
>>>> I might give it a go to try rewriting it tomorrow.
>>>
>>> http://dekoppel.eu/tmp/pedupgLarge.tar.gz (89 Mb)
>>>
>>> Contains two largish datasets in a directory structure 
>>> expected by the program.
>>
>> I only see 2 traits in that example, so it's hard for anyone 
>> to explore your scaling problem, seeing as there are a maximum 
>> of 2 tasks.
>
> Either way, a few small changes were enough to cut the runtime 
> by a factor of ~6 in the single-threaded case and improve the 
> scaling a bit, although the printing to output files still 
> looks like a bit of a bottleneck.
>

> http://dpaste.dzfl.pl/80cd36fd6796
>
> The key thing was reducing the number of allocations (more 
> std.algorithm.splitter copying to static arrays, less 
> std.array.split) and avoiding File.byLine. Other people in this 
> thread have mentioned alternatives to it that may be 
> faster/have lower memory usage, I just read the whole files in 
> to memory and then lazily split them with 
> std.algorithm.splitter. I ended up with some blank lines coming 
> through, so i added if(line.empty) continue; in a few places, 
> you might want to look more carefully at that, it could be my 
> mistake.
>
> The use of std.array.appender for `info` is just good practice, 
> but it doesn't make much difference here.

Wow, I'm impressed with the effort you guys (John, Rikki, others) 
are making to teach me some efficiency tricks. I guess this is 
one of the strengths of D: its community. I'm studying your 
various contributions closely!

The empty line comes from the very last line on the files, which 
also end with a newline (as per "normal" practice?).