"Spawn as many thousand threads as you like" and D

[Sönke Ludwig via Digitalmars-d]

Am 16.04.2014 20:34, schrieb Russel Winder via Digitalmars-d:
> On Wed, 2014-04-16 at 16:06 +0200, Sönke Ludwig via Digitalmars-d wrote:
> […]
>>
>> I agree, but I also wonder why you still keep ignoring vibe.d. It
>> achieves exactly that - right now! Integration with std.concurrency
>> would be great, but at least for now it has an API compatible
>> replacement that can be merged later when Sean's pull request is done.
>
> Vibe.d is a single-thread event system, which is great (*) for the sort
> of problems Node.js, Vert.x, Tornado, Flask, Sinatra, Ratpack are used
> for. The point here is that CSP and dataflow are a concurrency and
> parallelism model that D has not got.

I agree that those would be nice to have, but I think the main point was 
more about having transparent "green threads" than the actual 
concurrency model (at least that was the point that was itching me ;)

>
> std.concurrency is a heavyweight thread system so not really useful
> except to build thread pools and fork-join infrastructure. (OK that is a
> gross oversimplification.) std.parallelism is a great beginning of data
> parallelism on a thread pool. It needs more work. The thread pool needs
> to be brought front and centre, as a separate thing usable by other
> modules. On this CSP, dataflow, actors, etc. can be built.

At least the infrastructure part is pretty much in place for vibe.d when 
using worker tasks [1]. The tasks are getting distributed among a set of 
worker threads and fibers (fibers are getting reused for efficiency), so 
that spawning tasks is a very light-weight operation.

>
> Due to other commitments, not least leading a massive update of GPars, I
> cannot lead on working on D things. If however someone can drive, I will
> certainly contribute, along the lines as I did when David Simcha wrote
> std.parallelism – mostly as a tester and reviewer.
>
> This also raises the issue of the D infrastructure having an obvious and
> documented way for people to contribute to things like std.parallelism.
> Whatever the truth, the perception is that to work on something like
> std.parallelism, you have to fork the whole of Phobos. In fact,
> std.parallelism is a single file 4,500 lines long (**).
>
>
> (*) It would be even better if it supported mocking for unit tests ;-)

Absolutely, that's an issue I'm stumbling over every now and then for 
some parts of the code. The only issue is that it would need to be 
integrated in a way that doesn't make methods needlessly virtual (using 
interfaces everywhere) and doesn't break the API (using templates 
everywhere).

>
> (**) I am still not a fan of single files this big.
>

(I actually hate this. I like to have a mental model of the source code 
I'm working on. But for Phobos code the only possibility is usually to 
stab at local parts found via a full text search, which always leaves a 
bad taste when making changes due to the unknown implications. A proper 
hierarchical organization with defined dependencies could work wonders 
there.)

[1]: http://vibed.org/api/vibe.core.core/runWorkerTask