Profiling the memory in D

[Steven Schveighoffer]

On 12/4/19 5:04 PM, kerdemdemir wrote:

> GC.sizeof(cast(void*)object) will be super useful. I will use that.
> 
> I also tried GC: --DRT-gcopt=profile:1 already. It provides so little 
> information.
> I need to find out which member AA or array of which object is causing 
> this memory problem of mine.I am ending up around 2GB of ram usage in a 
> single day.

These are not easy to discover. Are you using 32-bit compiler? If so, it 
has an issue with false pointers -- basically a large piece of memory 
can get pinned by a non-pointer that happens to be on the stack. This is 
not so much an issue in 64-bit land because the chances of accidental 
pointers is infinitesimal.

> 
> Is there any way to manipulate profile-gc flag on run time? Like I will 
> start my program without it somehow and after the my program initializes 
> I will turn it on.

I'm not familiar much with the gc profiling features. Lately I have had 
to measure max GC size, which this does provide. Sorry I can't be more 
help there.

> 
> One last thing in my program I am getting a message from vibe sometimes 
> like
> "leaking eventcore driver because there are still active handles". I use 
> websockets and do web requests. I wanted to add that because I saw you 
> fixed something with Redis about that in 
> https://github.com/vibe-d/vibe.d/issues/2245.

What happens is that the eventcore driver provides a mechanism to 
allocate an implementation-specific chunk of data for each descriptor. 
This is done via C malloc. In the past, when the eventcore driver was 
shut down, these spaces were all deallocated when the driver was 
deallocated. If you had classes which held descriptors that were cleaned 
up by the GC, they would then try to use that space and generate a segfault.

So the eventcore driver will leak when it sees descriptors still in use, 
even when the GC is trying to clean it up because it doesn't know when 
the file descriptors will be released fully.

To fix, you need to ensure all file descriptors are destroyed 
deterministically. For the Redis problem, it was really difficult to 
solve without altering vibe.d itself, because the Redis session manager 
did not provide a mechanism to release all resources in the pool 
deterministically.

Tracking down what file descriptors are holding on, and who allocated 
them, is not an easy task. I had to instrument a lot of stuff to find 
it. Some easy possibilities may be if you aren't closing down all 
listening sockets before exiting main.

I will say that I still get these messages if I kill my web server while 
some keepalive connections are still open. But now, if there is no 
activity for about 10 seconds, then I get no messages.

Good luck tracking it down!

-Steve