std.concurrency and fibers

[Dmitry Olshansky]

On 05-Oct-12 08:27, Alex Rønne Petersen wrote:
> On 04-10-2012 22:04, Dmitry Olshansky wrote:

>> Cool, but currently it's a leaky abstraction. For instance if task is
>> implemented with fibers static variables will be shared among threads.
>> Essentially I think Fibers need TLS (or rather FLS) synced with language
>> 'static' keyword. Otherwise the whole TLS by default is a useless chunk
>> of machinery.
>
> Yeah, it's a problem all right. But we'll need compiler support for this
> stuff in any case.
>
> Can't help but wonder if it's really worth it. It seems to me like a
> simple AA-like API based on the typeid of data would be better -- as in,
> much more generic
> than trying to teach the compiler and runtime how
> to deal with this stuff.
> Think something like this:
>
> struct Data
> {
>      int foo;
>      float bar;
> }
>
> void myTask()
> {
>      auto data = Data(42, 42.42f);
>
>      TaskStore.save(data);
>
>      // work ...
>
>      foo();
>
>      // work ...
> }
>
> void foo()
> {
>      auto data = TaskStore.load!Data();
>      // work ...
> }
>
> I admit, not as seamless as static variables, but a hell of a lot less
> magical.

This just doesn't work though.
The true problem is not in the code you as a programmer doing distibuted 
stuff do.
It's library writers that typically use TLS for some persistent state 
inside module
and D currently makes it easy and transparent just like in the old non-MT
days but for threads ONLY.

Now having them all pack their stuff and go about fixing globals to 
TaskStore.store/.load
is not realistic and down right horrible.
Currently I suspect w.r.t. Fibers all that works is based on conventions 
& luck.

One problem with making everything FLS is that cost becomes darn high. 
On the other hand Fibers are yielded only manually (+scheduler now? 
probably on recive/send etc.) and a lot of things can be "fiber-safe" as is.

Also it seems like for this to work we need not only a scheduler but 
reworked libraries that are fiber-aware (so they don't block on I/O 
etc.). See e.g. vibe.d.

>>> C) We just swap out threads with fibers and document that the module
>>> uses fibers. See my comments in A for why I'm not sure this is a good
>>> idea.
>> Seems a lot like A but with task defined to be a fiber. I'd prefer this.
>> However then it needs a user-defined policy for distributing fibers
>> across real threads (pools). Btw A is full of this too.
>
> By choosing C we effectively give up any hope of distributed tasks and
> especially if we have a scheduler API. Is that really a good idea in
> this day and age?
>

Why? Remote fibers should go for a distributed tasks. Like I said just 
make Fiber == task.
As long as there is a suitable protocol for communication it's all 
right. I'm insisting on fiber as a task as this makes for simpler logic 
of message passing. And scheduler is still inevitable as fibers wait for 
messages and are multiplexed on only as many threads.

I just don't see any other abstraction you want to put in place of task. 
It should be self-contained persistent 'worker' so that message passing 
works transparently.


-- 
Dmitry Olshansky