TempAlloc (Was: Library Development: What to finish/flesh out?)

[dsimcha]

On 3/28/2011 7:38 PM, Masahiro Nakagawa wrote:
> On Sun, 27 Mar 2011 12:09:03 +0900, dsimcha <dsimcha at yahoo.com> wrote:
> 
>> On 3/18/2011 2:56 AM, Don wrote:
>>>> 3. TempAlloc: A memory allocator based on a thread-local segmented 
>>>> stack,
>>>> useful for allocating large temporary buffers in things like numerics
>>>> code.
>>>> Also comes with a hash table, hash set and AVL tree optimized for this
>>>> allocation scheme. The advantages over plain old stack allocation are
>>>> that
>>>> it's independent of function calls (meaning you can return pointers to
>>>> TempAlloc-allocated memory from a function, etc.) and it's segmented,
>>>> meaning
>>>> you can allocate huge buffers w/o risking stack overflow. Its main
>>>> weakness
>>>> is that this stack is not scanned by the GC, meaning that you can't
>>>> store the
>>>> only reference to a GC-allocated piece of memory here. However, in
>>>> practice
>>>> large arrays of primitives are an extremely common case in
>>>> performance-critical code. I find this module immensely useful in
>>>> dstats and
>>>> Lars Kyllingstad uses it in SciD. Getting it into Phobos would make it
>>>> easy
>>>> for other scientific/numerics code to use it. Completion state:
>>>> Working and
>>>> used. Needs a litte cleanup and documentation. (Phobos candidate)
>>>
>>> This is #1. Far and away. Belongs in druntime.
>>> I would use it instantly in BigInt.
>>
>> This is not up for "official" review yet because std.net.isemail and 
>> std.parallelism (and possibly more stuff depending on how we manage 
>> the review queue) are ahead of it. If you want a sneak preview:
>>
>> Code:
>>
>> https://github.com/dsimcha/TempAlloc
>>
>> Docs:
>>
>> http://cis.jhu.edu/~dsimcha/d/phobos/core_tempalloc.html
>>
>> I put in a few small, tangentially related memory allocation functions 
>> that I've wanted in druntime/Phobos for awhile in this proposal, too. 
>> alignedMalloc() and alignedFree() are used by TempAlloc but can be 
>> made private if people don't like them. newVoid() is not used by 
>> TempAlloc and might fit better in std.array, but I think it belongs 
>> somewhere in Phobos or druntime.
> 
> Now, I am thinking of light-weight process for concurrency.
> Your TempAlloc(segmented stack) seems to be a good parts for 
> implementing such feature :)
> 
> 
> Masahiro

I'm not sure what you're suggesting.  I'd definitely need more details
to help you in any way.  If it involves fibers, remember that
thread-local storage only works with OS threads.  TempAlloc stores its
state in thread-local storage.

Other than that, I'm interested in hearing about this.  If there are any
enhancements I can add to make what you want to do easier, let me know.