openMP

[dsimcha]

On Wednesday, 3 October 2012 at 21:02:07 UTC, David Nadlinger 
wrote:
> On Wednesday, 3 October 2012 at 19:42:07 UTC, dsimcha wrote:
>> If not, please clarify what you needed and the relevant use 
>> cases so that I can fix std.parallelism.
>
> In my use case, conflating the notion of a future, i.e. a value 
> that becomes available at some point in the future, with the 
> process which creates that future makes no sense.

So the "process which creates the future" is a Task that executes 
in a different thread than the caller?  And an alternative way 
that a value might become available in the future is e.g. if it's 
being retrieved from some slow I/O process like a database or 
network?

>
> For example, let's say you are writing a function which 
> computes a complex database query from its parameters and then 
> submits it to your query manager/connection pool/… for 
> asynchronous execution. You cannot use std.parallelism.Task in 
> this case, because there is no way of expressing the process 
> which retrieves the result as a delegate running inside a 
> TaskPool.

Ok, I'm confused here.  Why can't the process that retrieves the 
result be expressed as a delegate running in a TaskPool or a new 
thread?

>
> Or, say you want to write an "aggregator", combining the 
> results of several futures together, again offering the same 
> future interface (maybe an array of the original result types) 
> to consumers. Again, there is no computation-bound part to that 
> at all, which would make sense to run on a TaskPool – you are 
> only waiting on the other tasks to finish.

Maybe I'm just being naive since I don't understand the use 
cases, but why couldn't you just create an array of Task objects?

>
> The second problem with std.parallelism.Task is that your only 
> choice is polling (or blocking, for that matter). Yes, 
> callbacks are a hairy thing to do if you can't be sure what 
> thread they are executed on, but not having them severely 
> limits the power of your abstraction, especially if you are 
> dealing with non-CPU-bound tasks (as many of today's "modern" 
> use cases are).

I'm a little confused about how the callbacks would be used here. 
  Is the idea that some callback would be called when the task is 
finished?  Would it be called in the worker thread or the thread 
that submitted the task to the pool?  Can you provide a use case?

>
> For example, something my mentor asked to implement for Thrift 
> during last year's GSoC was a feature which allows to send a 
> request out to a pool of servers concurrently, returning the 
> first one of the results (apparently, this mechanism is used as 
> a sharding mechanism in some situations – if a server doesn't 
> have the data, it simply ignores the request).

"First one of the results" == the result produced by the the 
first server to return anything?

> How would you implement something like that as a function 
> Task[] -> Task? For what it's worth, Task in C# (which is quite 
> universally praised for its take on the matter) also has a 
> »ContinueWith« method which is really just a completion 
> callback mechanism.

I'll look into ContinueWith and see if it's implementable in 
std.parallelism without breaking anything.

>
> std.parallelism.Task is great for expressing local 
> resource-intensive units of work (and fast!), but I think it is 
> to rigid and specialized for that case to be generally useful.

Right.  I wrote std.parallelism with resource-intensive units of 
work in mind because that's the use case I was familiar with.  It 
was designed first and foremost to make using SMP parallelism 
_simple_.  In hindsight I might have erred to much on the side of 
making simple things simple vs. complicated things possible or 
over-specialized it and avoided solving the an important, more 
general problem.  I'll try to understand your use cases and see 
if they can be addressed without making simple things more 
complicated.

I think the best way you could help me understand what I've 
overlooked in std.parallelism's design is to give a quick n' 
dirty example of how an API that does what you want would be 
used.  Even more generally, any _concise, concrete_ use cases, 
even toy use cases, would be a huge help.