std.concurrency and efficient returns

[Robert Jacques]

On Tue, 03 Aug 2010 16:05:40 -0400, Sean Kelly <sean at invisibleduck.org>  
wrote:

> dsimcha Wrote:
>
>> == Quote from Sean Kelly (sean at invisibleduck.org)'s article
>> > I wouldn't worry--the modules in core will always provide the  
>> low-level control
>> that you're looking for.  As for message passing, I think it should be  
>> the default
>> choice but not the only choice.  Things like futures are a reasonable  
>> addition to
>> Phobos as well.
>>
>> How explicit do we want to make inter-thread sharing when using a
>> futures/tasks/parallel foreach library?  This question may be a result  
>> of the fact
>> that I (still) don't understand what shared is supposed to do, but do  
>> we want to
>> force all data to be shared/cast to shared even under a  
>> futures/tasks/parallel
>> foreach model, or do we want to bypass the shared system entirely and  
>> simply stick
>> to old-fashioned "here be dragons" when doing future/task/parallel  
>> foreach-based
>> multithreading?
>
> Ideally, the parallelism should be both invisible to the user and  
> automatically safe.  That isn't a realistic goal though, so a balance  
> must be found between performance and safety.  I'd venture to say the  
> "here be dragons" approach is correct for the most part though.  If too  
> much performance is traded away for safety, no one will use it.

My experience with data-parallel programming leads me to believe that a  
large number of use cases could be covered by extending the D's set of  
function/member modifiers (i.e. const/shared/immutable/pure) to cover  
delegates. This would allow, for instance, a parallel foreach function to  
take a const delegate or a future function to take a shared delegate and  
thereby provide both safety and performance. Bartosz recently bloged about  
task driven parallelism in three "High Productivity Computing Systems"  
languages ( Chapel, X10, Fortress ) and criticized all three regarding  
taking the "here be dragons" approach. Even experts wake up the dragons on  
a semi-regular basis and debugging them back into slumber is both hard and  
time consuming; it's typically to see a 10x cost in development time on  
massively parallel architectures. For mere mortals to be productive, those  
dragons need to be slain. And I think the D has a decent chance to be the  
first(?) non-functional language to slay them. But I think this should be  
a goal of D3, and that a "here be dragons" library for D2 would be  
extremely useful: because if we're going to slay the dragons, we need to  
know as about many caves and dragons as possible.