Parallelizing code -- design problem in networkish code

Charles Hixson charleshixsn at earthlink.net
Thu Dec 13 13:32:02 PST 2012


I'm trying to parallelize some code which is essentially a network of 
cells with an index.  I can make the cells immutable, if I route all 
access to them via the index, AND if I can update the index.  The index 
would not need to have items deleted, but updates would cause it to 
point to different addresses as new versions of the cells replaced old 
versions.  Also the index is LARGE.  I can't size it exactly, but it 
would have over 100,000 entries.  To be safe I've said it's indexed by a 
ulong, though I don't have enough RAM to allow that to be reasonable.

So now I'm trying to parallelize it.  I don't want a per thread copy for 
multiple reasons (it's large, it gets updates, etc.).  But if I read 
TDPL correctly, shared won't allow me to update it.  If I can't update 
it, I need to restart the program to resize it AND the cells can't be 
immutable.

It would appear that making the cells immutable is silly.  All I need is 
to protect a few small places where I adjust various weights.  But TDPL 
says that if I adopt that approach, I also need write locks around every 
read.  So that looks infeasible.

I need the index mainly to sequentially step through the cells, so I 
could implement an equivalent functionality by having prior and next 
pointers in each cell...but if I use pointers then making one cell 
immutable makes the whole network immutable, and that's no good at all.

Is there a decent answer in D?  This is basically one class replicated a 
huge number of times.  Ideally each instance would be in a separate 
thread, but that would overwhelm the OS, and I don't have more than 
around 6 cores on my current system.  But if each instance were in a 
separate thread, then I could do this with message passing....except for 
stepping through all the cells sequentially. (Well, I could do that by 
implementing prior thread and next thread variables, I guess, but that 
seems contrary to the purpose of threads.  Still, that's a mandatory 
part of the problem.)


More information about the Digitalmars-d-learn mailing list