How many std.concurrency receivers?

[thedeemon]

On Thursday, 11 October 2012 at 16:09:20 UTC, Charles Hixson 
wrote:

> Hmmm...what I'm trying to build is basically a cross between a 
> weighted directed graph and a neural net, with some features of 
> each, but not much in common.  Very light-weight processes 
> would be ideal.  The only communication should be via 
> message-passing.  Each cell would spend most of it's time 
> sitting on a count-down timer waiting to be rolled out to a 
> database of inactive processes, but it needs to maintain local 
> state (weights of links, activation level, etc.  nothing fancy.)
>
> If I were doing this sequentially, I'd want to use structs for 
> the cells, because class instances would be too heavy.  And I'd 
> store them in a hash table keyed by cell-id#.
>
> Unfortunately, I don't see any reasonable way of chunking the 
> pieces, so that I can chunk them into 100 relatively 
> independent sets.  Or even 1000.  10,000 is probably about the 
> right size for active-at-one-time cells.  And if it would 
> handle that, std.concurrency seemed ideal.
>
> Do you have any suggestions as to what would be a reasonable 
> better choice?  (Outside of going back to sequential.)

Here's how I would try to approach a task of having thousands of 
independent agents with current std.concurrency. Each agent 
(cell) is represented by some data structure and its main 
function which gets one message as input, reacts (possibly 
changing its state and sending other messages) and returns 
without blocking. Then I'd create say 16 threads (or 8, anyway a 
power of 2 which is close to actual number of cores), each of 
them will have its own message queue, that's given by 
std.concurrency. Let's say each cell has its own id. I would 
place cell with id N to the thread number N mod 16. Each thread 
will have an array of cells mapped to it. Then if some cell sends 
a message to cell X, it makes sure the message contains cell id 
of recipient and then sends it to thread X mod 16. Each worker 
thread runs a loop where it receives next message from its queue, 
finds the target cell by its id in this thread's array of cells 
(we can use X / 16 as index) and calls its reaction function. 
This way all agents are evenly distributed between threads, we're 
using just 16 threads and 16 queues which work in parallel, and 
it all acts as if thousands of agents work independently. However 
this approach does not guarantee even work distribution between 
cores.