Why does this simple test program leak 500MB of RAM?

[Steven Schveighoffer]

On 6/19/23 12:40 PM, FeepingCreature wrote:

> Some additional info I probably should have mentioned.
> 
> Note that I'm manually calling the GC at the end. At that point, 
> everything should be dead. heaptrack shows that we get something like 8 
> big GC runs over the program runtime: as expected, because it's 
> overprovisioning a "healthy" amount.
> 
> My standing theory is that it's the zombie stack issue again: see 
> https://forum.dlang.org/post/befrzndhowlwnvlqcoxx@forum.dlang.org for 
> what I think is happening.
> 
> I'm currently experimenting with setting up a fiber, then zeroing out 
> its stack completely after the function has run in it. Results are 
> mixed, but promising. I still end up with ~100MB live at the end though 
> - which admittedly is a lot better than 600MB.

Are you looking at GC memory usage or process usage? Because the GC 
doesn't generally give back memory to the OS even when it could. And 
fragmentation can stop it from doing so also -- an entire allocated pool 
has to be unallocated for it to release it to the OS.

I know there has always been mysterious "reasons" for the GC to not 
collect things that are stack-referenced, but I tend to first try to 
rule out looking at the wrong information, and/or other simple 
explanations before getting there.

Having somewhat recently looked at how the GC scanning works for 
threads, I'm reasonably certain that unused stack isn't being scanned 
(even if it's allocated). On a 64-bit machine, the likelihood of 
accidentally keeping things referenced on the stack is low.

I would also point out that your specific case allocates *most* of the 
memory secondary referenced from the stack. That is, you might refer to 
the AA structure from the stack, but not the bucket elements (where the 
bulk is allocated). And using `aa.clear` should make that even less 
possible. All of the allocated buckets should be definitively garbage 
with no possibility of zombie references.

-Steve