[phobos] Phobos unit tests running out of memory

[Rainer Schuetze]

On 05.09.2012 00:39, Rainer Schuetze wrote:
> On 05.09.2012 00:07, Andrei Alexandrescu wrote:
>> Interesting. Can we fix that?
>
> If I read the dmc runtime library sources correctly (heap32/heap.cpp),
> heap allocations work with a minimum chunk size of 64 kB but
> unfortunately 4 bytes get added to the requested size before calling
> VirtualAlloc. That actually blocks 128kB of virtual address space, but
> only 64k+4 bytes are used.
>
> So, I recommend just aligning to the next multiple of 64kB in
> RTLHeap::MoreCore. Walter, can you rebuild snn.lib with an appropriate fix?

I patched snn.lib to that effect, i.e. replaced this code

   if (Size < 0x10000)
     Size = 0x10000;
   else
     Size += Size/5;  // Allocate 20% more to eliminate fragmentation

with this:
   Size = ((Size + 4 + Size/5 + 0xffff) & ~0xffff) - 4;
   // 4 is added in call to sbrk

and ran this little test program:

import core.stdc.stdlib;
import core.stdc.stdio;

void main()
{
	for(int i = 0; ; i++)
		if(malloc(1024) is null)
		{
			printf("allocated %d MB\n", i/1024);
			break;
		}
}

Compiling with the original snn.lib and running under Win8/64bit, it 
outputs 962 MB of allocated memory, 1954 MB if I set the 
large-address-aware bit.

With the patch, it can allocate 1925 MB and 3906 MB, respectively.

It takes minutes to run this little program (no swapping involved, I 
have 8GB of memory), slowing down to allocating about 5MB/s when 
watching progress in the process explorer. That seems to suggest that 
the heap function do not deal well with that amount of memory or at 
least with this allocation pattern.

If you want to try it yourself, start your favorite hex editor, load 
snn.lib, find

81 FE 00 00 01 00 73 07 BE 00 00 01 00 EB 0D 8B C6 BB 05 00 00 00 31 D2 
F7 F3 01 C6 8D 56 04

and replace it with

89 f0 BB 05 00 00 00 31 D2 F7 F3 8D 84 30 03 00 01 00 25 00 00 FF FF 8D 
70 FC 90 90 8D 56 04

Rainer