TempAlloc: Request for Review For Inclusion in core.memory

[Daniel Gibson]

Am 19.04.2011 17:02, schrieb Denis Koroskin:
> On Tue, 19 Apr 2011 18:34:33 +0400, dsimcha <dsimcha at yahoo.com> wrote:
> 
>> == Quote from Andrej Mitrovic (andrej.mitrovich at gmail.com)'s article
>>> Nice work. I've tried it out with allocating some large float arrays,
>>> quick two test cases were a 94MB and 282MB audio file.
>>> File 1 w/o TempAlloc: 410msecs
>>> File 1 w/ TempAlloc: 184msecs
>>> File 2 w/o TempAlloc 1321 msecs
>>> File 2 w/ TempAlloc 567 msecs
>>> Btw, how much *can* I allocate at once by using new (not specific to
>>> TempAlloc)? I've tried a 500MB file, but allocation fails even though
>>> I have plenty of free memory on my system. Is there a limit to using
>>> 'new'?
>>> Of course, ideally I'd use some kind of buffering mechanism where a
>>> portion of a file is loaded into memory, and when that's exhausted I'd
>>> load another chunk. But I'd like to know the limits for a single
>>> memory allocation on the garbage-collected heap. Is there some
>>> internal limit for D? I know I can't allocate more than 3GB in the
>>> entire app, but I'm looking at single allocations.
>>> Pardon me if this all sounds stupid, I'm a complete newb when it comes
>>> to memory allocation mechanics. :)
>>
>> Ironically, these benchmarks just show that the C heap is faster than
>> the D GC
>> heap.  They are well over the 4 MB mark, where TempAlloc just falls
>> back on the C
>> heap.  As far as how much you can allocate on the GC heap, it boils
>> down to how
>> much **contiguous** free space you have in your address space.  Since
>> 32-bit
>> address space is only 4 GB (and only 2 GB for user space on Win32),
>> I'm not
>> surprised you can't allocate more than 500 MB.
> 
> I think you are wrong about contiguous memory block. I believe virtual
> address space is divided into blocks (which are, indeed, contiguous) but
> the address space itself is not. What I mean is that if you try to
> allocate 1GB of memory, an OS might succeed that even if the physical
> memory is "fragmented" by other processes.
> 

That doesn't help if the logical memory of your process is fragmented as
well :)

> This isn't the case for ALL operating systems, though.
> 
> I've believe some OSes don't even allocate memory physically until you
> actually access it (by reading or writing to it), and even then they may
> only allocate memory for requested pages only.
> 
> I might be wrong though.