std.concurrency and fibers

[Alex Rønne Petersen]

On 05-10-2012 04:14, deadalnix wrote:
> Le 04/10/2012 13:32, Alex Rønne Petersen a écrit :
>> Hi,
>>
>> We currently have std.concurrency as a message-passing mechanism. We
>> encourage people to use it instead of OS threads, which is great.
>> However, what is *not* great is that spawned tasks correspond 1:1 to OS
>> threads. This is not even remotely scalable for Erlang-style
>> concurrency. There's a fairly simple way to fix that: Fibers.
>>
>> The only problem with adding fiber support to std.concurrency is that
>> the interface is just not flexible enough. The current interface is
>> completely and entirely tied to the notion of threads (contrary to what
>> its module description says).
>>
>> Now, I see a number of ways we can fix this:
>>
>> A) We completely get rid of the notion of threads and instead simply
>> speak of 'tasks'. This trivially allows us to use threads, fibers,
>> whatever to back the module. I personally think this is the best way to
>> build a message-passing abstraction because it gives enough transparency
>> to *actually* distribute tasks across machines without things breaking.
>> B) We make the module capable of backing tasks with both threads and
>> fibers, and expose an interface that allows the user to choose what kind
>> of task is spawned. I'm *not* convinced this is a good approach because
>> it's extremely error-prone (imagine doing a thread-based receive inside
>> a fiber-based task!).
>> C) We just swap out threads with fibers and document that the module
>> uses fibers. See my comments in A for why I'm not sure this is a good
>> idea.
>>
>> All of these are going to break code in one way or another - that's
>> unavoidable. But we really need to make std.concurrency grow up; other
>> languages (Erlang, Rust, Go, ...) have had micro-threads (in some form)
>> for years, and if we want D to be seriously usable for large-scale
>> concurrency, we need to have them too.
>>
>> Thoughts? Other ideas?
>>
>
> Something I wonder for a while : why not run everything in fibers ?

Because then we definitely need dynamic stack growth wired into both the 
compiler and the runtime.

Not impossible, but there's a *lot* of effort required (and convincing, 
in Walter's case).

-- 
Alex Rønne Petersen
alex at lycus.org
http://lycus.org