Any advice for GC/memory debugging - tricks (phase 1)

[Alan Knowles]

Ok  - adding the minimize code from tango into phobos seems ok - it does 
reduce the pools...

However - Its not really a fix, but a bandaid to a worse problem.. 
(leaky code...)

It took me quite a while to track down the major causes of my memory 
leaks, here are some of the tricks (which would be really useful if they 
where in phobos/tango core..)

setMaxAlloc() - ability to set the maximum memory that alloc can be 
called with. - and just return null if it's more than that..
- Why? - well, quite a common error is
a ~= ..... (where a is an ever growing string) -- this can quickly eat 
up pages of memory, and is quite difficult to track down. - forcing a 
segfault by not allowing memory allocation, makes finding the cause with 
backtrace in gdb very simple.

allocLogOn() ...  this would help alot - seeing that you are allocating 
alot of small buffers eg. this code

char[] recv() {
    ubyte[] buf  = new ubyte[1024];
    numb = stream.read(buf);
    ...
}
which eats up memory very quickly - is better done using a class 
property.... eg.
ubyte[] buf  = new ubyte[1024];
char[] recv() {
    numb = stream.read(buf);
    ...
}


I'm going to look at thread registering of memory (eg. having an 
associative array of memusage[threadid] = size) so it's easier to work 
out where a thread might be leaking memory, even after a fullCollect().





Regards
Alan




Alan Knowles wrote:
> Having fleshed out an SMTP server, I've now started testing it against a 
> good load level.
> 
> Apart from the various thread issues in phobos (getHostbyName is not 
> thread safe etc./ using udns solved alot of this...) - my bigger worry 
> is memory management.
> 
> With GC enabled/or even disabled, the memory pool will rise to ~ 200Mb 
> from the baseline of ~20K and run out of memory on the box it's running 
> on. The test box we have peaks at about 30 concurrent 
> connections/threads. - compiled using GDC (as it's FreeBSD)
> 
>  From what i've seen, there is no real way to disable the Garbage 
> collection, in terms of stopping it from managing the 
> allocation/deallocation of memory? as it is too intertwined in the 
> management of things like char[] etc..
> 
> Has anyone found good ways to  analyze where the application may be 
> leaking memory/allocating to much memory - hooks in the in/out 
> properties of methods ?
> 
> Is there any way to compact the memory, as the pool appears to jump 
> quite dramatically from ~ 64k->100Mb->250Mb, even though it looks like 
> the used pool is significantly smaller..
> 
> I'm basically fishing for ideas here, as I've got a whole day tomorrow 
> to start sticking in debugging code to find some answers...
> 
> Regards
> Alan