Where is my memory?

[Rikki Cattermole via Digitalmars-d]

On 22/03/2015 10:42 p.m., "Ozan =?UTF-8?B?U8O8ZWwi?= 
<ozan.sueel at gmail.com>" wrote:
> Hi!
> I'm working on a Big Data project, where a huge amount of RAM is needed.
> Using D I've run into a - let's called it - memory leak. I tested this
> with following code:
>
>      foreach(i; 0..1000) {
>          int[] ints;
>          foreach(j; 0..1000) {
>              ints ~= uniform(0, 100);
>          }
>      }
>
> Without the first foreach this code use only 220KB,
> with the first foreach this code needs 2,5MB.
> (220KB x 1000 is something around 2,5MB).
>
> But why is GC (Garbage Collector) not running? Following the
> explanations in http://wiki.dlang.org/Memory_Management memory usage
> should be something around  220KB.
>
> I used GC.minimize, slow down the loop, replaced uniform...doesn't work.
> (By the way: After a while my LINUX server killed my big data project
> because running out of RAM & SWAP space)
>
>
> Thanks for your advice,
> Ozan

Since you know exact sizes perhaps, a rewrite is needed?

int[] buffer;
buffer.length = 1000 * 1000;

size_t offset;
foreach(i; 0 .. 1000) {
	foreach(j; offset .. offset + 1000) {
		buffer[j] = uniform(0, 100);
	}	

	offset += 1000;
}

Really that buffer should be malloc'ed without the GC knowing about it. 
And later manually free'd.

In fact, I would recommend moving that buffer as:

static int[1000 * 1000] buffer;

void myfunc() {
	size_t offset;
	foreach(i; 0 .. 1000) {
		foreach(j; offset .. offset + 1000) {
			buffer[j] = uniform(0, 100);
		}	

		offset += 1000;
	}
	// use buffer
}

This method will not allocate per execution. But be careful. Each 
running of myfunc will overwrite what is in buffer.