Debugging memory leak.

[Bill Baxter]

Bill Baxter wrote:
> Frits van Bommel wrote:
>> David Brown wrote:
>>> On Mon, Oct 08, 2007 at 10:23:17PM -0700, David Brown wrote:
>>>
>>>> I seem to have something else in my program that is making lots of 
>>>> stray
>>>> pointers for the GC to follow.  My program consistently leaks memory 
>>>> until
>>>> it is killed by running out of address space (or I kill it because 
>>>> it is
>>>> trying to swap my machine to death).
>>>
>>> I think I found my problem.  I have something like this:
>>>
>>>   class Foo {
>>>     ubyte[20] hash;
>>>     ...
>>>   }
>>>
>>> and a _lot_ of these are alive at any given time.  The hash is a sha1 
>>> hash,
>>> and so tends to be evenly distributed.
>>>
>>> It appears that heap objects only have a single flag indicating 
>>> whether or
>>> not they have pointers, and since classes do have pointers, the GC will
>>
>> Yeah, it's just a single bit currently. The vtable and monitor 
>> pointers don't count though (the first points to static memory and the 
>> second to malloc()ed (non-gc) memory), so class bodies only gets 
>> tagged as containing pointers if it has explicit 
>> pointer/reference/dyn. array/assoc.array members.
>>
>>> look at the "pointers" in the hash to see if they hit anything.  
>>> Build up a
>>> reasonable set of these and something will point to almost everything.
>>>
>>> It's going to take me a little while to change the code to not do 
>>> this (it
>>> isn't quite as simple as my example).  I was concerned about having 20
>>> bytes living in its own gc allocation being wasteful, but leaking nearly
>>> everything is much more wasteful :-)
>>
>> A precise GC (or at least something more in that direction) would 
>> really help in these kinds of situations...
> 
> I think there's another big potential source of leaks:  Associative Arrays.
> 
> A node in an AA has two pointers and a hash value, so the gc will treat 
> all those hash values as pointers too.  I think that's probably what's 
> killing me, because I use a lot of AA's.
> 
> --bb


Here's a simple test case:

module memunion;
import std.stdio;
import std.gc;

// Just an ordinary AA with a lot of values.
class BigAA
{
     int[int] aa;
     this() {
         for(int i=0;i<1000;i++) {
             aa[i] = i;
         }
     }
}

void main()
{
     int nloops = 10_000;
     auto b = new BigAA[100];

     for(int i=0; i<nloops; ++i)
     {
         // Create some AAs (overwriting old ones)
         foreach(ref v; b) { v = new BigAA; }

         // See how we're doing
         std.gc.GCStats stats;
         std.gc.fullCollect();
         std.gc.getStats(stats);
         writefln("Loop %-5s - poolsize=%-10s   %s Mbytes  (%s KB)",
                  i, stats.poolsize,
                  stats.usedsize/1024/1024,
                  stats.usedsize/1024);

     }
}


With phobos, mem usage goes up pretty steadily.

--bb