Memory issues. GC not giving back memory to OS?

[ikod]

On Tuesday, 21 April 2020 at 18:31:28 UTC, Cristian Becerescu 
wrote:
> Hi!
>
> A little bit of context first:
>
> I was using DPP and I noticed huge amounts of RAM being used.
> So I used valgrind massif and found out that 98% of the 
> process’ memory (~6GB) was allocated for arrays / Appender with 
> mmap.
>
> I then performed a simple test where I incrementally appended 
> 2^30 integers (4GB) to a dynamic array (memory measurements are 
> the same for Appender).
> -> Memory used (peak; increasing towards the end of execution): 
> ~7GB
> -> capacity == 1.107 * size (at the end of the program)
>
> This is a bit odd, because 1.107 * 2^30 is roughly 4.4GB, and 
> the peak memory consumption was 7GB. Apparently, the GC can 
> correctly collect the memory when manually calling collect() at 
> the end of appending, but that memory (we are talking 7 - 4.4 = 
> 2.6GB) is never given back to the system. At least this is our 
> intuition after making those observations.
>
> I have created a gist with the test code and results (thanks 
> Edi for augmenting the test code to profile the GC): 
> https://gist.github.com/cbecerescu/e6606a8530c56ae06c52e5b1cd32b31f
>
> Just some notes:
> - if reserving 2^30 elements for the array (or Appender) 
> beforehand, memory peaks are at 4GB
> - C++'s std::vector, without reservation, never gets beyond 4GB 
> and has size == capacity at the end

IMHO this happens because each time you requested larger 
contiguous memory region. Runtime have to allocate (or 
reallocate) larger piece of memory (at higher addresses), copy 
old content and then release old piece of memory. But old piece 
of memory can't be released to OS as heap area can be released 
only from the top.