GC memory fragmentation

[tchaloupka]

Hi,
we're using vibe-d (on Linux) for a long running REST API server 
and have problem with constantly growing memory until system 
kills it with OOM killer.

First guess was some memory leak so we've added periodic call to:

```D
GC.collect();
GC.minimize();
malloc_trim(0);
```

And when called very often (ie 500ms) service memory stays stable 
and doesn't grow, so it doesn't look as a memory leak (or at 
least not that fast to explain that service can go from 90MB to 
900MB in 2 days).

But this is very bad for performance so we've prolonged the 
interval for example to 30 seconds and now memory still goes up 
(not that dramatically but still).

Stats of the GC when it growed up between 30s are these (logged 
after `GC.collect()` and `GC.minimize()`:

```
GC.stats: usedSize=9994864, freeSize=92765584, total=102760448
GC.stats: usedSize=11571456, freeSize=251621120, total=263192576
```

Before the grow it has 88MB free space from 98MB total allocated.
After 30s it has 239MB free from 251MB allocated.

So it wastes a lot of free space which it can't return back to OS 
for some reason.

Can these numbers be caused by memory fragmentation? There is 
probably a lot of small allocations (postgresql query, result 
processing and REST API json serialization).

Only explanation that makes some sense is that in some operation 
there is required memory allocations that can't be fulfilled with 
current memory pool (ie due to the memory fragmentation in it) 
and then it allocates some data in new memory segment that can't 
be returned afterwards as it still holds the 'live' data. But 
that should be freed too at some point and GC should minimize (if 
another request doesn'cause allocation in the 'wrong' page again).

But still, the amount of used vs free memory seems wrong, its a 
whooping 95% of free space that can't be minimized :-o. I have a 
problem imagining some fragmentation in it.

Are there any tools that can help diagnosing this more?

Also note that `malloc_trim` is not called from the GC itself and 
as internally used malloc handles it's own memory pool it has 
it's own quirks with returning unused memory back to the OS (it 
does it only on `free()` in some cases).
See for example: 
http://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html

Behavior of malloc can be controlled with `mallopt` with:

```
M_TRIM_THRESHOLD
     When  the  amount  of contiguous free memory at the top of 
the heap grows sufficiently large, free(3) employs sbrk(2) to 
release this memory back to the system.  (This can be useful in 
programs that continue to execute
     for a long period after freeing a significant amount of 
memory.)  The M_TRIM_THRESHOLD parameter specifies the minimum 
size (in bytes) that this block of memory must reach before 
sbrk(2) is used to trim the heap.

     The default value for this parameter is 128*1024.
```

But the default is 128kB free heap memory block for trim to 
activate but when `malloc_trim` is called manually, a much larger 
block of memory is often returned. Thats puzzling too :)