Asynchronicity and more

[Fawzi Mohamed]

There are several difficult issues connected with asynchronicity, high  
performace networking and connected things.
I had to deal with them developing blip ( http://fawzi.github.com/ 
blip ).
My goal with it was to have a good basis for my program dchem, and as  
consequence is not so optimized in particular for non recursive tasks,  
and it is D1, but I think that the issues are generally relevant.

i/o and asynchronicity is a very important aspect and one that will  
tend to "pollute" many parts of the library, and introduce  
dependencies that are difficult to remove thus those choices have to  
be done carefully.

Overview:
========

Threads vs fibers:
-----------------------

* an issue not yet brought up is that thread wire some memory, and so  
have an extra cost that fibers don't.
* evaluation strategy of fibers can be chosen by the user, this is  
relevant for recursive tasks where each task
   spawns other tasks, different strategies (breadth first evaluation  
like threads uses a *lot* more resources
   than depth first, by having many more tasks concurrently in  
evaluation)

Otherwise the relevant points already  brought forth by others are:

- context switch of fibers (assuming that memory is active) is much  
faster
- context switch are chosen by the user in fibers (cooperative  
multitasking), this allows
   one to choose the optima point to switch, but a "bad" fibers can  
ruin the response time the others.
- d is not stackless (like Go for example), so each fiber needs to  
have enough space for the stack
   (something that often is not so easy to predict). This makes fiber  
still a bit costly if one really needs a lot of them.
   64 bit can help here, because hopefully the active part is small,  
and it can be kept in RAM, even using a rather
   large virtual space. Still as correctly said by Brad for heavily  
uniform handling of many tasks manual
   management (and using stateless functions as much as possible) can  
be much more efficient.

Closures
------------
When possible and for the low level (often used) operations delegates  
and functions calls are a better solution than , structs and manual  
memory handling for "closures" are a good choice for low level  
operations, because one can avoid the heap allocation connected with  
the automatic closure.
This approach cannot be avoided in D1, whereas D2 has the very useful  
closures, but at low level their cost should be avoided when possible.
About using structs there are subtle issues that I think are connected  
with optimization of the compiler (I never really investigated them, I  
always changed the code, or resorted to heap allocation.
The main issue is that one would like to optimize as much as possible,  
and to do it it normally assumes that the current thread is the only  
user of the stack. If you pass stack stored structures to other  
threads these assumptions aren't true anymore, so the memory of a  
stack allocated struct might be reused even before the function  
returns (unless I am mistaken and the ABI forbids it, in this case  
tell me).

Async i/o
----------

* almost always i/o is much slower than CPU, so an i/o operation is  
bound to make the cpu wait, so one wants to use the wait efficiently.
   - A very simple way is to just use blocking i/o, and just have  
other threads do other threads.
   - async i/o allows overlap of several operations in a single thread.
   - for files an even more efficient way to communicate sharing of  
the buffer with the kernel (aio_*)
   - an important issue is avoiding waste of cpu cycles while waiting,  
to achieve this one can collect several waiting operations and use a  
single thread to wait on several of them, select, poll and epoll allow  
this, and increase the efficiency of several kinds of programs
   - libev and libevent are cross platform libraries that can help  
having an event based approach, taking care to check a large number of  
events and call a user defined callback when they happen in a robust  
cross platform way

locks, semaphores
------------
to synchronize between threads locks and semaphores are a standard way  
to synchronize.
One has to be careful to mix them with fiber scheduling with locks, as  
one can easily deadlock.

Hardware informationy
-----------------------------
Efficient usage of computational resource depends also on being able  
to identify the available hardware.
Don did quite some hacking to get useful information out of cpuinfo,  
but if one is interested in more complex computers more info would be  
nice.
I use hwloc for this purpose, it is cross plattform, can be embedded.

Possible solutions
==============

Having async i/o can be presented as normal synchronous (blocking) i/ 
o, but this makes sense only if one has several objects waiting, or  
uses fibers, and executes other fiber while waiting.
How acceptable it is to rely (and thus introduce a dependency on)  
things like libev or hwloc?
For my purposes using them was ok, and they are cross platform and  
embeddable, but is it true also for phobos?

Asynchronicity means being able to have work to be executed  
concurrently and then resynchronize at a later point.
One can use processes (that also give memory protection), threads, or  
fibers to achieve this.
If one uses just threads, then asynchronous i/o makes sense only with  
a fully manual (explicit) handling of it, hiding it away will be  
equivalent to blocking i/o.
Fibers allow one to hide async io and make it look as blocking, but as  
Sean told there are issues with using fibers with D2 TLS.
I kind of dislike the use of TSL for non low level infrastructure  
stuff, but that is just me around here it seems.

In blip I choose to go with fiber based switching.
I wrapped libev both at low level and at a higher level, in such a way  
than one can use them directly (for maximum performance)
For the sockets I use non blocking calls, and a single "waiting" (io)  
thread, but hide them so that they are used just like blocking calls.

An important design decision if using fibers is if one should be able  
to have a "naked" thread, or hide the fiber scheduling in each thread.
In blip I went for yes, because it is entirely realized as a normal  
library, but that gives some ugly corner cases when one uses a method  
that wants to suspend a thread that doesn't  have scheduling place.
Building the scheduling into all threads is probably cleaner if one  
goes with fibers.
The problem of TSL and fibers remains though, especially if one allows  
the migration of fibers from one thread to the other (as I do in blip).

An important design choice in blip was being able to cope with  
recursive parallelism (typical of computation tasks), not just with  
the (server like) concurrent parallelism that is typical of servers.
I feel that it is important, but is something that might not be seen  
as such by others.

To do
====
Now about async io the first step is for sure to expose an  
asynchronous API. This doesn't influence or depends on other parts of  
the library much.
An important decision if/which external libraries one can rely on.

Making the async API nicer to use, or even use it "behind the scenes"  
as I do in blip needs more complex choices on the basic handling of  
suspension and synchronization.
Something like that is bound to be used in several parts of phobos so  
a careful choice is needed.

This parts are also partially connected with high performance  
networking (another GSoC project).

Fawzi

Fawzi