[phobos] Parallelism in Phobos

[Michel Fortin]

Le 2010-09-10 à 17:13, David Simcha a écrit :

> As far as I can tell, your needs might be better served by std.concurrency.

From what I can see, your parallel foreach is basically some syntactic sugar for queuing tasks inside a loop and then to block until the result is ready. While I'll admit I'm not sure I need that sugar or to block waiting for the result, queuing tasks in a loop is certainly something I need.

With my app I can easily have 1000 of these tasks queued at a given time (I effectively have a couple of loops that can add tasks to a queue). They mostly read and parse files to extract some pieces of data. At the API level, std.concurency looks like it could do that, except it'd be creating one thread for each task. I don't want to create one thread for each task, so I need some sort of task queue and a thread pool.

But maybe you're right, and maybe the thread pool should go in std.concurrency where creating and queuing a task could work like spawning a thread, perhaps like this:

	// send task to a specific thread to be executed there
	tid.perform(&taskFunc, "hello world");

	// queue task for execution in a thread pool
	tpool.dispatch(&taskFunc, "hello world");

Those two things I'd find quite useful. And it'd be pretty much trivial to build a parallel foreach on top of this.

And just to add weight to the argument that task based concurrency is used pretty much everywhere: I worked before on some industrial software that had this too. It basically had to perform some analysis every time new data came in, in real-time. A new task was created for each piece of data and dispatched to a thread pool, then a few seconds later the result was sent to another thread that'd take some action based on the analysis.

-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/