std.container.BinaryHeap + refCounted = WTF???

[Steven Schveighoffer]

On Wed, 17 Nov 2010 10:14:21 -0500, dsimcha <dsimcha at yahoo.com> wrote:

> == Quote from Steven Schveighoffer (schveiguy at yahoo.com)'s article
>> I think that we need a wrapper for containers that implements the shared
>> methods required and manually locks things in order to use them.  Then  
>> you
>> apply this wrapper to any container type, and it's now a shared  
>> container.
>> There are also certain types of containers that lend themselves to  
>> shared
>> access.  For example, I can see a linked list where each node contains a
>> lock being a useful type.
>
> This is a good idea to some degree, but the thing is that it forces you  
> to use
> shared even when you're going for fine-grained parallelism and want to  
> just cowboy
> it.  For fine-grained parallelism use cases, my hunch is that cowboying  
> is going
> to be the only game in town for a long time in all languages, not just D.

There is always the possibility of "cowboy"ing it.  But I don't see that  
the standard lib should be catering to this.

>
>> > (For arrays, I'm referring to the appending issue, which is  
>> problematic
>> > when I try
>> > to append to an array from multiple threads, synchronizing manually.)
>> I'm interested what you mean here, I tried to make sure cross-thread
>> appending is possible.
>> >> dcollections containers would probably all fail if you tried to use  
>> them
>> >>  from multiple threads.
>
> Ok, I stand corrected.  It seemed to work in practice, but always I just  
> assumed
> that it was a Bad Thing to do and worked for the Wrong Reasons.

There is specific code in array appending that locks a global lock when  
appending to shared arrays.  Appending to __gshared arrays from multiple  
threads likely will not work in some cases though.  I don't know how to  
get around this, since the runtime is not made aware that the data is  
shared.

>> memory (a relatively abundant resource).
>
> Apparently you've never tired working with multigigabyte datasets using a
> conservative garbage collector and 32-bit address space.

Is that supported by out-of-the-box containers?  I would expect you need  
to create a special data structure to deal with such things.

And no, I don't regularly work with such issues.  But my point is,  
reference counting the *container* which uses some sort of memory  
allocation to implement its innards is not coupling a limited resource to  
memory allocation/deallocation.  In other words, I think it's better to  
have the container be a non-reference counted type, even if you  
reference-count the elements.  I prefer class semantics to be quite  
honest, where explicit initialization is required.

-Steve