TempAlloc: Request for Review For Inclusion in core.memory

[Denis Koroskin]

On Tue, 19 Apr 2011 18:34:33 +0400, dsimcha <dsimcha at yahoo.com> wrote:

> == Quote from Andrej Mitrovic (andrej.mitrovich at gmail.com)'s article
>> Nice work. I've tried it out with allocating some large float arrays,
>> quick two test cases were a 94MB and 282MB audio file.
>> File 1 w/o TempAlloc: 410msecs
>> File 1 w/ TempAlloc: 184msecs
>> File 2 w/o TempAlloc 1321 msecs
>> File 2 w/ TempAlloc 567 msecs
>> Btw, how much *can* I allocate at once by using new (not specific to
>> TempAlloc)? I've tried a 500MB file, but allocation fails even though
>> I have plenty of free memory on my system. Is there a limit to using
>> 'new'?
>> Of course, ideally I'd use some kind of buffering mechanism where a
>> portion of a file is loaded into memory, and when that's exhausted I'd
>> load another chunk. But I'd like to know the limits for a single
>> memory allocation on the garbage-collected heap. Is there some
>> internal limit for D? I know I can't allocate more than 3GB in the
>> entire app, but I'm looking at single allocations.
>> Pardon me if this all sounds stupid, I'm a complete newb when it comes
>> to memory allocation mechanics. :)
>
> Ironically, these benchmarks just show that the C heap is faster than  
> the D GC
> heap.  They are well over the 4 MB mark, where TempAlloc just falls back  
> on the C
> heap.  As far as how much you can allocate on the GC heap, it boils down  
> to how
> much **contiguous** free space you have in your address space.  Since  
> 32-bit
> address space is only 4 GB (and only 2 GB for user space on Win32), I'm  
> not
> surprised you can't allocate more than 500 MB.

I think you are wrong about contiguous memory block. I believe virtual  
address space is divided into blocks (which are, indeed, contiguous) but  
the address space itself is not. What I mean is that if you try to  
allocate 1GB of memory, an OS might succeed that even if the physical  
memory is "fragmented" by other processes.

This isn't the case for ALL operating systems, though.

I've believe some OSes don't even allocate memory physically until you  
actually access it (by reading or writing to it), and even then they may  
only allocate memory for requested pages only.

I might be wrong though.