Memory allocation in D

[Sean Kelly]

Marius Muja wrote:
> Sean Kelly wrote:
> 
>> For what it's worth, my concern with the current behavior is that its 
>> worst-case scenario coincides with typical usage.  It is extremely 
>> common for a programmer to use array sizes that are powers of two, and 
>> this is the situation which results in the maximum unused space per 
>> block (because the runtime adds 1 to the size of all array allocations 
>> passed to the GC).  Totally stinks, and I'm running out of ideas.  I 
>> don't suppose someone can offer a suggestion for how to address this, 
>> other than "just always allocate exactly what they request and if they 
>> get a page fault then too bad for them?"
> 
> That was my problem also. I was using array sizes that were powers of 
> two and because of the extra 1 there were many memory pages wasted. For 
> example in one of my tests I was allocating float arrays of size 1024 
> (4096 bytes) which were taking 2 pages instead of 1 because of the extra 
> 1 added, and that's where I was observing that with D's new operator I 
> can only allocate half as much memory than with malloc.
> Oddly the same problem occurs if I allocate much larger arrays of 
> exactly 1024*1024 floats (4MB size). Seems that instead of 4MB, D's 
> runtime allocated 8MB for each of these arrays. Is D's runtime using 4MB 
> pages for large memory allocations?

Weird.  D's GC is basically a "big bag of pages" allocator, where groups 
of pages are divided into pools.  The maximum size a pool will be on its 
own is 8 megs, with very large single allocations getting their own pool 
that is as big as necessary.  If I had to guess, I'd say the 8MB you're 
seeing is an artifact of this allocation scheme, and I'd suspect that 
much of the 8MB is committed but marked as free pages in your program.


Sean