problem with parallel foreach
Gerald Jansen via Digitalmars-d-learn
digitalmars-d-learn at puremagic.com
Tue May 12 13:50:44 PDT 2015
On Tuesday, 12 May 2015 at 17:45:54 UTC, thedeemon wrote:
> On Tuesday, 12 May 2015 at 17:02:19 UTC, Gerald Jansen wrote:
>
>> About 3.5 million lines read by main(), 0.5 to 2 million lines
>> read and 3.5 million lines written by runTraits (aka runJob).
>
> Each GC allocation in D is a locking operation (and disabling
> GC doesn't help here at all), probably each writeln too, so
> when multiple threads try to write millions of lines such issue
> is easy to meet. I would look for a way to write those lines
> without allocations and locking, and also reduce total number
> of system calls by buffering data, doing less f.writef's.
Your advice is appreciated but quite disheartening. I was hoping
for something (nearly) as easy to use as Python's parallel.Pool()
map(), given that this is essentially an "embarassingly parallel"
problem. Avoidance of GC allocation and self-written buffered IO
functions seems a bit much to ask of a newcomer to a language.
More information about the Digitalmars-d-learn
mailing list