GC memory fragmentation

[tchaloupka]

On Monday, 12 April 2021 at 07:03:02 UTC, Sebastiaan Koppe wrote:
>
> We have similar problems, we see memory usage alternate between 
> plateauing and then slowly growing. Until it hits the 
> configured maximum memory for that job and the orchestrator 
> kills it (we run multiple instances and have good failover).
>
> I have reduced the problem by refactoring some of our gc usage, 
> but the problem still persists.
>
> On side-note, it would also be good if the GC can be aware of 
> the max memory it is allotted so that it knows it needs to do 
> more aggressive collections when nearing it.

I knew this must be a more common problem :)

What I've found in the meantime:

* nice writeup of how GC actually works by Vladimir Panteleev - 
https://thecybershadow.net/d/Memory_Management_in_the_D_Programming_Language.pdf
   * described tool (https://github.com/CyberShadow/Diamond) would 
be very helpfull, but I assume it's for D1 and based on some old 
druntime fork :(
* we've implemented log rotation using `std.zlib` (by just 
`foreach (chunk; fin.byChunk(4096).map!(x => c.compress(x))) 
fout.rawWrite(chunk);`)
   * oh boy, don't use `std.zlib.Compress` that way, it allocates 
each chunk and for a large files it creates large GC memory peaks 
that sometimes doesn't go down
   * rewritten using direct `etc.c.zlib` completely out of GC
* currently testing with `--DRT-gcopt=incPoolSize:0` as otherwise 
allocated page size multiplies with number of allocated pools * 
3MB by default
* `profile-gc` is not much helpfull in this case as it only 
prints total allocated memory for each allocation on the 
application exit and as it's a long running service using many 
various libraries it's just hundreds of lines :)
   * I've considered to fork the process periodically, terminate 
it and rename the created profile statistics to at least see the 
differences between the states, but still not sure if it would 
help much
* as I understand the GC it uses different algorithm for small 
allocations and for large objects
   * small (`<=2048`)
     * it categorizes objects to fixed set of used sizes and for 
each uses whole memory page as bucket with free list from which 
it reserves memory on request
     * when the bucket is full, new page is allocated and 
allocations are provided from that
   * big - similar, but it allocates N pages as a pool

So If I understand it correctly when for example vibe-d 
initializes new fiber on some request, it's handled and fiber can 
be discarded it can easily lead to a scenario when fiber itself 
is allocated in one page, it's filled up during the request 
processing so new page is allocated and when cleaning, bucket 
with fiber cannot be cleaned up as it's added to a `TaskFiber` 
pool (with a fixed size). This way fiber's bucket would never be 
freed and easily never be used anymore during the application 
lifetime.

I'm not so sure if pages of small objects (or large) that are not 
completely empty can be reused as a new bucket or only free pages 
can be reused.

Does anyone has some insight of this?

Some kind of GC memory dump and analyzer tool as mentioned 
`Diamond` would be of tremendous help to diagnose this..