[Issue 4566] New: Calling functions in parallel with std.concurrency

[d-bugmail at puremagic.com]

http://d.puremagic.com/issues/show_bug.cgi?id=4566

           Summary: Calling functions in parallel with std.concurrency
           Product: D
           Version: D2
          Platform: Other
        OS/Version: Linux
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: Phobos
        AssignedTo: nobody at puremagic.com
        ReportedBy: jmdavisProg at gmail.com


--- Comment #0 from Jonathan M Davis <jmdavisProg at gmail.com> 2010-08-01 17:03:55 PDT ---
Okay. I'm finding that it's a common pattern with my threads that I'm
effectively trying to call a function, have it run in a separate thread, and
then have the result returned to the calling thread (which at present is done
through send/receive). Essentially, I end up with something like this in the
calling thread

auto tid1 = spawn(&func1, /* args */);
auto tid2 = spawn(&func2, /* args */);
auto tid3 = spawn(&func3, /* args */);
T1 retValFromTid1;
T2 retValFromTid2;
T3 retValFromTid3;
receive(/* whatever is appropriate to receive sends from funcs */);
receive(/* whatever is appropriate to receive sends from funcs */);
receive(/* whatever is appropriate to receive sends from funcs */);
/* retValFromTidXs should now all have the values sent from the spawned threads
*/


and something like this in each spawned thread:

void func1(Tid parentTid, /* args */)
{
    /* obviously, in non-array cases, something other than idup has to be done
to make the data immutable */
    send(parentTid, thisTid, callToActualFunc(/* args */).idup);
}


This causes 3 problems, 2 of which I think are solvable and one of which
probably isn't.

1. The args passed to the spawned thread are often intended solely for that
thread and never used again in the calling thread. Having to copy mutable data
to make it immutable to pass it along to spawn() and then possibly having to
copy it _again_ in the spawned thread to make it mutable again is not
particularly efficient and definitely annoying. Ideally, there would be a way
to indicate that ownership of the data is being handed over to the spawned
thread and the calling thread would no longer have access to it.

2. Having to copy the data returned from the spawned thread in order to make it
immutable, only to have to copy it again on the receiving end to make it
mutable again is also painfully ineffecient. And unless you use something like
Rebindable on the other end, you _have_ copy it to make it mutable again - even
if you could have used it as immutable - because the local variables that
you're assigning it to can't be immutable because you're setting them in
receive() rather then when they're declared.

3. Having to use spawn(), receive(), and send() in this manner (not to mention
possibly having to create a wrapper function just to make the function being
called a thread and pass its return value back) is quite messy in comparison to
just calling a function. There has got to be a cleaner way to do this. It's
great if you intend to continually pass messages back and forth, but for the
case where you're basically trying to call several unrelated functions in
parallel and get their return values, this is messy.

With regards to #1, I'm not sure that it can be fixed, because it very quickly
seems like it becomes an issue similar to escaped references. You have to have
a way to ensure that the calling thread doesn't hold onto references to the
data being passed (or at least that it doesn't access them before the spawned
threads have terminated). I'd love for it to be fixed, but I'm not sure that it
can be.

With regards to #2 and #3, I think that they're totally fixable with a new set
of functions in std.concurrency specifically for this case. That is you have a
function which you pass a function to call, the arguments for that function,
and the variable which the function's return value will be assigned to - and it
takes multiple such functions at a time (that may require creating a struct to
hold the set of data for each function, but that wouldn't necessarily be a
problem). And that function call would block until it had received the return
values for each of those functions and set them to the variables which were
given for that purpose. The functions being called would be normal functions
with return values. Due to #1, they'd probably have to take immutable reference
types - though they might be able to take const because the calling thread
would be blocking until they terminated, and even if the same data were passed
to each of the threads, it would have to be const for all of them, so they
might be able to get away with const parameters rather than immutable. However,
they would return as normal in either case, so you could genarally avoid
wrapper functions, and you wouldn't have to make the return value immutable
(since you _know_ that the thread returning it can't alter it anymore, since
it's terminating).

Essentially, you'd get something like this:

auto f1 = ThreadedFunc!(&func1, /* args */);
auto f2 = ThreadedFunc!(&func1, /* args */);
auto f3 = ThreadedFunc!(&func1, /* args */);
callThreadedFuncs(f1, f2, f3);
/* f1.retval, f2.retval, and f3.retval have the returned values */


I don't know if that's the best way to handle this scenario, but it seems to me
that it would be of great value to have a way in phobos to simultaneously call
a set of functions (waiting for all of them to finish before moving on) where
the number of copies due to forced immutability is minimized if not eliminated.
As it is, it's rather cumbersome and inefficent to use std.concurrency to
effectively call multiple functions in parallel.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------