Where is my memory?

[Steven Schveighoffer via Digitalmars-d]

On 3/22/15 6:32 AM, anonymous wrote:
> On Sunday, 22 March 2015 at 09:42:41 UTC, Ozan Süel wrote:
>> Hi!
>> I'm working on a Big Data project, where a huge amount of RAM is
>> needed. Using D I've run into a - let's called it - memory leak. I
>> tested this with following code:
>>
>>     foreach(i; 0..1000) {
>>         int[] ints;
>>         foreach(j; 0..1000) {
>>             ints ~= uniform(0, 100);
>>         }
>>     }
>>
>> Without the first foreach this code use only 220KB,
>> with the first foreach this code needs 2,5MB.
>> (220KB x 1000 is something around 2,5MB).
>
> 220KB x 1000 would be around 200MB.
>
>> But why is GC (Garbage Collector) not running? Following the
>> explanations in http://wiki.dlang.org/Memory_Management memory usage
>> should be something around  220KB.
>
> The GC is running. 1000 x 1000 ints would be 4MB. That alone is more
> than the 2.5MB you're observing.
>
>> I used GC.minimize, slow down the loop, replaced uniform...doesn't work.
>> (By the way: After a while my LINUX server killed my big data project
>> because running out of RAM & SWAP space)
>
> GC.minimize alone can't do anything if the memory isn't collected. Use
> GC.collect(), too.
>
> Also, if you know the final size of the array beforehand, you can
> `reserve` it. That avoids a lot of copying (and garbage creation) when
> appending.
>
>          int[] ints;
>          ints.reserve(1000); /* <- */
>          foreach(j; 0..1000) {
>              ints ~= uniform(0, 100);
>          }

Another tip, don't throw away that memory you allocated!

int[] ints;
ints.reserve(1000);
foreach(i; 0..1000)
{
    ints.length = 0;
    ints.assumeSafeAppend;
    ... // inner loop
}

-Steve