Simple parallel foreach and summation/reduction

[Chris Katko]

On Monday, 24 September 2018 at 05:59:20 UTC, Chris Katko wrote:
> On Saturday, 22 September 2018 at 02:26:41 UTC, Chris Katko 
> wrote:
>> On Saturday, 22 September 2018 at 02:13:58 UTC, Chris Katko 
>> wrote:
>>> On Friday, 21 September 2018 at 12:15:59 UTC, Ali Çehreli 
>>> wrote:
>>>> On 09/21/2018 12:25 AM, Chris Katko wrote:
>>>>> [...]
>>>>
>>>> You can use a free-standing function as a workaround, which 
>>>> is included in the following chapter that explains most of 
>>>> std.parallelism:
>>>>
>>>>   http://ddili.org/ders/d.en/parallelism.html
>>>>
>>>> That chapter is missing e.g. the newly-added fold():
>>>>
>>>>   
>>>> https://dlang.org/phobos/std_parallelism.html#.TaskPool.fold
>>>>
>>>> Ali
>>>
>>> Okay... so I've got it running. The problem is, it uses tons 
>>> of RAM. In fact, proportional to the working set.
>>>
>>> T test(T)(T x, T y)
>>> 	{
>>> 	return x + y;
>>> 	}
>>> 	
>>> double monte(T)(T x)
>>> 	{
>>> 	double v = uniform(-1F, 1F);
>>> 	double u = uniform(-1F, 1F);
>>> 	if(sqrt(v*v + u*u) < 1.0)
>>> 		{
>>> 		return 1;
>>> 		}else{
>>> 		return 0;
>>> 		}
>>> 	}
>>>
>>> 	auto taskpool = new TaskPool();
>>> 	sum = taskpool.reduce!(test)(
>>> 	taskpool.amap!monte(
>>> 		iota(num)
>>> 		)	);	
>>> 	taskpool.finish(true);
>>>
>>> 1000000 becomes ~8MB
>>> 10000000 becomes 80MB
>>> 100000000, I can't even run because it says "Exception: 
>>> Memory Allocation failed"
>>
>> Also, when I don't call .finish(true) at the end, it just sits 
>> there forever (after running) like one of the threads won't 
>> terminate. Requiring a control-C. But the docs and examples 
>> don't seem to indicate I should need that...
>
> So I looked into it. It's amap that explodes in RAM.
>
> Per the docs, amap has "less overhead but more memory usage." 
> While map has more overhead but less memory usage and "avoids 
> the need to keep all results in memory."
>
> But, if I make a call to map... it doesn't compile! I get:
>
>     Error: no [] operator overload for type 
> std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
>
> Simply changing amap to map here:
>
>     sum = taskPool.reduce!(test)
>         (
>         taskPool.map!(monte)(range)
>         );


Actually, I just realized/remembered that the error occurs inside 
parallelism itself, and MANY times at that:

/usr/include/dmd/phobos/std/parallelism.d(2590): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2596): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
/usr/include/dmd/phobos/std/parallelism.d(2634): Error: no [] 
operator overload for type 
std.parallelism.TaskPool.map!(monte).map!(Result).map.Map
monte.d(64): Error: template instance 
std.parallelism.TaskPool.reduce!(test).reduce!(Map) error 
instantiating

Though I tried looking up the git version of prallelism.d and the 
lines don't quite line up:

https://github.com/dlang/phobos/blob/master/std/parallelism.d