review of std.parallelism

[dsimcha]

On 3/19/2011 1:09 PM, Michel Fortin wrote:
> On 2011-03-19 12:03:51 -0400, Andrei Alexandrescu
> <SeeWebsiteForEmail at erdani.org> said:
>
>> * "Most of this module completely subverts..." Vague characterizations
>> ("most", "completely", "some") don't belong in a technical
>> documentation. (For example there's either subversion going on or
>> there isn't.) Also, std.concurrency and std.parallelism address
>> different needs so there's little competition between them. Better:
>> "Unless explicitly marked as $(D @trusted) or $(D @safe), artifacts in
>> this module are not provably memory-safe and cannot be used with
>> SafeD. If used as documented, memory safety is guaranteed."
>
> Actually, I think this is a bad description of what it subverts. What it
> subverts isn't the memory-safety that SafeD provides, but the safety
> against low-level races that even unsafe D protects against unless you
> cast shared away. For instance:
>
> void main() {
> int sum = 0;
> foreach (int value; taskPool.parallel([0,2,3,6,1,4,6,3,3,3,6])) {
> sum += value;
> }
> writeln(sum);
> }
>
> The "+=" would need to be an atomic operation to avoid low-level races.
>
> I think that ParallelForeach's opApply should only accept a shared
> delegate. I define shared delegate as a delegate that does not reference
> any non-shared variables of its outer scope. The problem is that DMD
> currently doesn't know how to determine whether a delegate literal is
> shared or not, thus a delegate literal is never shared and if
> ParallelForeach's opApply asked a shared delegate as it should it would
> just not work. Fix DMD to create shared delegate literals where
> appropriate and everything can be guarantied race-free.
>

If you want pedal-to-metal parallelism without insane levels of 
verbosity, you can't have these safety features.  I thought long and 
hard about this issue before submitting this lib for review and 
concluded that any solution would make std.parallelism so slow, so 
limited and/or such a PITA to use that I'd rather it just punt these 
issues to the programmer.  In practice, parallel foreach is used with 
very small, performance-critical snippets that are fairly easy to reason 
about even without any language-level race safety.  This is a 
fundamental design decision that will not be changing.  If it's 
unacceptable then:

1.  I give up.

2.  I wish someone had told me earlier.