Why does this simple test program leak 500MB of RAM?

[FeepingCreature]

On Monday, 19 June 2023 at 17:38:17 UTC, Steven Schveighoffer 
wrote:
> On 6/19/23 12:40 PM, FeepingCreature wrote:
>
>> Some additional info I probably should have mentioned.
>> 
>> Note that I'm manually calling the GC at the end. At that 
>> point, everything should be dead. heaptrack shows that we get 
>> something like 8 big GC runs over the program runtime: as 
>> expected, because it's overprovisioning a "healthy" amount.
>> 
>> My standing theory is that it's the zombie stack issue again: 
>> see 
>> https://forum.dlang.org/post/befrzndhowlwnvlqcoxx@forum.dlang.org for what I think is happening.
>> 
>> I'm currently experimenting with setting up a fiber, then 
>> zeroing out its stack completely after the function has run in 
>> it. Results are mixed, but promising. I still end up with 
>> ~100MB live at the end though - which admittedly is a lot 
>> better than 600MB.
>
> Are you looking at GC memory usage or process usage? Because 
> the GC doesn't generally give back memory to the OS even when 
> it could. And fragmentation can stop it from doing so also -- 
> an entire allocated pool has to be unallocated for it to 
> release it to the OS.
>
> I know there has always been mysterious "reasons" for the GC to 
> not collect things that are stack-referenced, but I tend to 
> first try to rule out looking at the wrong information, and/or 
> other simple explanations before getting there.
>
> Having somewhat recently looked at how the GC scanning works 
> for threads, I'm reasonably certain that unused stack isn't 
> being scanned (even if it's allocated). On a 64-bit machine, 
> the likelihood of accidentally keeping things referenced on the 
> stack is low.
>
> I would also point out that your specific case allocates *most* 
> of the memory secondary referenced from the stack. That is, you 
> might refer to the AA structure from the stack, but not the 
> bucket elements (where the bulk is allocated). And using 
> `aa.clear` should make that even less possible. All of the 
> allocated buckets should be definitively garbage with no 
> possibility of zombie references.
>
> -Steve

I'm looking at process usage. But I mean, the thing is, I *can* 
make it go down to sub 100MB by playing with the stack.

I'll post code for that version when I get to work.