Some missing things in the current threading implementation

[Sönke Ludwig]

>> 1. spawn and objects
>> 	Spawn only supports 'function' + some bound parameters. Since taking
>> the address of an object method in D always yields a delegate, it is not
>> possible to call class members without a static wrapper function. This
>> can be quite disturbing when working object oriented (C++ obviously has
>> the same problem).
>
> Except in the case of an immutable or shared object this would be unsafe, as it
> would allow implicit sharing.  I do agree, though, that delegates need to be
> allowed if they're immutable or shared delegates.  Right now taking the address of
> a shared/immutable member function doesn't yield a shared/immutable delegate.
> There are bug reports somewhere in Bugzilla on this.
>

Good to know that there are already bug reports. I remember the 
discussion about allowing shared(delegate) or immutable(delegate) and 
this would be a possible solution. However, I still find the idea that 
those attributes are bound to the delegate type awkward and wrong, as a 
delegate is typically supposed to hide away the internally used 
objects/state and this is just a special case for direct member function 
delegates (what about an inline (){ obj.method(); }?). Also const is not 
part of the delegate and does not have to be because it can be checked 
at delegate creation time. But this is probably a topic on its own.

>> 2. error messages
>> 	Right now, error messages just state that there is a shared/unshared
>> mismatch somewhere. For a non-shared-expert, this can be a real bummer.
>> You have to know a lot of implications 'shared' has to be able to
>> correctly interpret these messages and track down the cause. Not very
>> good for a feature that is meant to make threading easier.
>
> Agreed.  Whenever you run into an unreasonably obtuse error message, a bug report
> would be appreciated.  Bug reports related to wrong or extremely obtuse error
> messages are considered "real", though low priority, bugs around here.
>

I will definitely file bug reports when I continue in this area, I just 
wanted to stress how important the error messages are in this part of 
the language, because the root cause is most often very non-obvious 
compared to other type-system errors.

>> 4. steep learning curve - more a high learning wall to climb on
>> 	Resulting from the first points, my feeling tells me that a newcomer,
>> who has not followed the discussions and thoughts about the system here,
>> will see himself standing before a very high barrier of material to
>> learn, before he can actually put anything of it to use. Also I imagine
>> this to be a very painful process because of all the things that you
>> discover are not possible or those error messages that potentially make
>> you banging your head against the wall.
>
> True, but I think this is just a fact of life when dealing with concurrency in
> general.  Gradually (partly due to the help of people like you pointing out the
> relevant issues) the documentation, etc. will improve.

...plus that even for someone who is already experienced with threading 
in other langugages, there is a lot to know now in D if you go the 
shared path instead of the C++/__gshared path.

>
>> 5. advanced synchronization primitives need to be considered
>> 	Things such as core.sync.condition (the most important one) need to be
>> considered in the 'shared'-system. This means there needs to be a
>> condition variable that takes a shared object instead of a mutex or you
>> have to be able to query an objects mutex.
>
> The whole point of D's flagship concurrency model is that you're supposed to use
> message passing for most things.  Therefore, lock-based programming is kind of
> half-heartedly supported.  It sounds like you're looking for a low-level model
> (which is available via core.thread and core.sync, though it isn't the flagship
> model).  std.concurrency is meant to be a high-level model useful for simple, safe
> everyday concurrency, not the **only** be-all-and-end-all model of multithreading
> in D.
>
>> 6. temporary unlock
>> 	There are often situations when you do lock-based programming, in which
>> you need to temporarily unlock your mutex, perform some time consuming
>> external task (disk i/o, ...) and then reaquire the mutex. For this
>> feature, which is really important also because it is really difficult
>> and dirty to work around it, needs language support, could be something
>> like the inverse of a synchronized {} scope or the possibility to define
>> a special kind of private member function that unlocks the mutex. Then,
>> inside whose blocks the compiler of course has to make sure that the
>> appropriate access rules are not broken (could be as conservative as
>> disallowing access to any class member).
>
> Again, the point of std.concurrency is to be primarily message passing-based.  It
> really sounds like what you want is a lower-level model.  Again, it's available,
> but it's not considered the flagship model.
>

Agreed that the flagship model is message passing and to a degree I 
think that is quite reasonable (except that object orientation + message 
passing comes a bit too short IMO). However, I think the support for the 
rest is a bit too half hearted if you have to use casts for everything. 
There are quite some low hanging fruits where a simple syntax or library 
extension could increase the flexibility without sacrificing safety or 
complexity.

>> 9. unique
>
> Just tested this, and it doesn't compile.
>

Forgot the 'shared' in that example:

---
import std.concurrency;

synchronized class Test {
	void publicMethod(){
		spawn( function void(shared Test inst){ inst.privateMethod(); }, this );
	}
	
	private void privateMethod(){
	}
}
---

>> 	II. Implementation of a ThreadPool
>
> My std.parallelism module that's currently being reviewed for inclusion in Phobos
> has a thread pool and task parallelism, though it is completely unsafe (i.e. it
> allows implicit sharing and will not be allowed in @safe code).  std.concurrency
> was simply not designed for pull-out-all-stops parallelism, and pull-out-all-stops
> parallelism is inherently harder than basic concurrency to make safe.  I've given
> up making most std.parallelism safe, but I think I may be able to make a few
> islands of it safe.  The question is whether those islands would allow enough
> useful things to be worth the effort.  See the recent safe asynchronous function
> calls thread.  Since it sounds like you need something like this, I'd sincerely
> appreciate your comments on this module.  The docs are at:
>
> http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html
>
> Code is at:
>
> http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d
>
>
>> 	III. multiple threads computing separate parts of an array
>
> Also in the proposed std.parallelism module, though completely unsafe because it
> needs to be fast.
>

I will definitely be looking into std.parallelism (I already have a 
thread pool, but that right now is not really sophisticated, mostly 
because of the previous lack of some advanced synchronization primitives).