std.parallelism equivalents for posix fork and multi-machine processing

[via Digitalmars-d]

On Thursday, 14 May 2015 at 20:28:20 UTC, Laeeth Isharc wrote:
> My own is a pragmatic commercial one.  I have some problems 
> which perhaps scale quite well, and rather than write it using 
> fork directly, I would rather have a higher level wrapper along 
> the lines of std.parallelism.

Languages like Chapel and extended versions of C++ have built in 
support for parallel computing that is relatively effortless and 
designed by experts (Cray/IBM etc) to cover common patterns in 
demanding batch processing for those who wants something higher 
level than plain C++ (or in this case D which is pretty much the 
same thing).

However, you could consider combining single threaded processes 
in D with e.g. Python as a supervising process if the datasets 
allow it. You'll find lots of literature on Inter Process 
Communication (IPC) for Unix. Performance will be lower, but your 
own productivity might be higher, YMMV.

> Perhaps such would be flawed and limited, but often something 
> is better than nothing, even if not perfect.  And I mention it 
> on the forum only because usually I have found the problems I 
> face turn out to be those faced by many others too..

You need momentum in order to get from a raw state to something 
polished, so you essentially need a larger community that both 
have experience with the topic and a need for it in order to get 
a sensible framework that is maintained.

If you can get away with it, the most common simplistic approach 
seems to be map-reduce. Because it is easy to distribute over 
many machines and there are frameworks that do the tedious bits 
for you.

> If you have any thoughts on what should be considered, I would 
> very much appreciate them.  (And I owe you a response on our 
> last suspended discussion, but haven't had time of late).

Nah, you owe me nothing ;-). And I also have no time atm. ;-)

Ola.