Simple parallel foreach and summation/reduction
Chris Katko
ckatko at gmail.com
Mon Sep 24 07:13:24 UTC 2018
On Monday, 24 September 2018 at 05:59:20 UTC, Chris Katko wrote:
> On Saturday, 22 September 2018 at 02:26:41 UTC, Chris Katko
> wrote:
>> On Saturday, 22 September 2018 at 02:13:58 UTC, Chris Katko
>> wrote:
>>> On Friday, 21 September 2018 at 12:15:59 UTC, Ali Çehreli
>>> wrote:
>>>> On 09/21/2018 12:25 AM, Chris Katko wrote:
>>>>> [...]
>>>>
>>>> You can use a free-standing function as a workaround, which
>>>> is included in the following chapter that explains most of
>>>> std.parallelism:
>>>>
>>>> http://ddili.org/ders/d.en/parallelism.html
>>>>
>>>> That chapter is missing e.g. the newly-added fold():
>>>>
>>>>
>>>> https://dlang.org/phobos/std_parallelism.html#.TaskPool.fold
>>>>
>>>> Ali
>>>
>>> Okay... so I've got it running. The problem is, it uses tons
>>> of RAM. In fact, proportional to the working set.
>>>
>>> T test(T)(T x, T y)
>>> {
>>> return x + y;
>>> }
>>>
>>> double monte(T)(T x)
>>> {
>>> double v = uniform(-1F, 1F);
>>> double u = uniform(-1F, 1F);
>>> if(sqrt(v*v + u*u) < 1.0)
>>> {
>>> return 1;
>>> }else{
>>> return 0;
>>> }
>>> }
>>>
>>> auto taskpool = new TaskPool();
>>> sum = taskpool.reduce!(test)(
>>> taskpool.amap!monte(
>>> iota(num)
>>> ) );
>>> taskpool.finish(true);
>>>
>>> 1000000 becomes ~8MB
>>> 10000000 becomes 80MB
>>> 100000000, I can't even run because it says "Exception:
>>> Memory Allocation failed"
>>
>> Also, when I don't call .finish(true) at the end, it just sits
>> there forever (after running) like one of the threads won't
>> terminate. Requiring a control-C. But the docs and examples
>> don't seem to indicate I should need that...
>
> So I looked into it. It's amap that explodes in RAM.
>
> Per the docs, amap has "less overhead but more memory usage."
> While map has more overhead but less memory usage and "avoids
> the need to keep all results in memory."
>
> But, if I make a call to map... it doesn't compile! I get:
>
> Error: no [] operator overload for type
> std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
>
> Simply changing amap to map here:
>
> sum = taskPool.reduce!(test)
> (
> taskPool.map!(monte)(range)
> );
Actually, I just realized/remembered that the error occurs inside
parallelism itself, and MANY times at that:
/usr/include/dmd/phobos/std/parallelism.d(2590): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2596): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2634): Error: no []
operator overload for type
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
monte.d(64): Error: template instance
std.parallelism.TaskPool.reduce!(test).reduce!(Map) error
instantiating
Though I tried looking up the git version of prallelism.d and the
lines don't quite line up:
https://github.com/dlang/phobos/blob/master/std/parallelism.d
More information about the Digitalmars-d-learn
mailing list