parallelFuture

[Charles Hixson]

dsimcha wrote:
> I've created an alpha release of parallelFuture, a high-level parallelization
> library for D2.  Right now, it has a task pool, futures, parallel foreach, and
> parallel map.
> 
> Here's the (IMHO) coolest example:
> 
> auto pool = new ThreadPool();
> 
> // Assuming we have a function isPrime(), print all
> // prime numbers from 0 to uint.max, testing for primeness
> // in parallel.
> auto myRange = iota(0, uint.max);
> foreach(num; pool.parallel(myRange)) {
>     if(isPrime(num)) {
>         synchronized writeln(num);
>     }
> }
> 
> The interface is there and it seems to work, although it has not been
> extensively stress tested yet.  Some of the implementation details could
> admittedly use some cleaning up, and I would appreciate help from some
> threading gurus on improving my queue (right now it's a naive synchronized
> singly-linked list) and getting condition mutexes to work properly.  (Right
> now, I'm using atomic polling followed by sleeping for 1 millisecond in a lot
> of places.  It's a kludge, but it seems to work reasonably well in practice.)
> 
> The code is at:
> 
> http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/parallelFuture.d
> 
> The docs are at:
> 
> http://cis.jhu.edu/~dsimcha/parallelFuture.html

If you can easily do it, it would be nice to be able to have the threads 
able to communicate with each other.  Something along the lines of both 
broadcast messages and 1:1 exchanges.  A very rough sketch of a possible 
  use:
Task[] tasks  =  Task.who_is_waiting;
foreach (auto task; tasks)
{  if (task.process(something) )
    {  task.markDone(true);   }
}

I see "something" as being an Object that would need to implement an 
interface to carry some identifying information, so when a task received 
it it could determine what to do with it (with "reject processing" being 
an option).  I think the rest is pretty clear.

OTOH, this comment was just to demonstrate the kind of communication 
between tasks that I'm talking about.  Static methods for broadcast 
communication and instance methods for 1:1 communication.  With 
safeties, so when a task completes before it gets your message, you 
aren't left talking to a null pointer.  Cleanup would need to be managed 
by the original thread that created the tasks.  It would need to be able 
to send a broadcast "now closing task xxx signal" to all threads. 
Threads maintaining a list of tasks would need to scan through them and 
remove any references to that task.

Maybe this is getting to complicated, but it would certainly be useful. 
  In the past interthread communication is the main problem that's kept 
me from using them.