A program leaking memory.

Johan Grönqvist johan.gronqvist at gmail.com
Mon Mar 13 11:43:10 PST 2006


Thomas: Thak you very much for your answer!

Now I have found the cause to this problem,

The garbage collector collects so rarely that my system starts swapping
intensely.

This time I only used gdc.
(gcc version 4.0.1, gdc 0.17, using dmd 0.140)

Any hints to the solution are most welcome!

Is there a way to tell the gc never to use more than (say) 100MB of memory?
I could insert an if-clause in each step, calling the collector if some
limit is exceeded, but if feels a bit inelegant.


Comments to Thomas Kuehnes very helpful answer below.

/ johan




1) Static vs. Dynamic arrays:
--------------------------------------------------------
The system size I use will not be known at compile time, and my
understanding is that I can therefore not use

real[myLength] grad;

but have to use

real[] grad;
grad.length = myLength;

where myLength is computed earlier.

The arrays I initialize in this way are never again resized. With this I
mean that the two first things I do are

real[] grad;
grad.lenth = myLength;

thereafter I modify the elements and read off their values a lot, but I
never resize the array again, i.e., the memory need for grad should
never change during a step, and as soon as one step ends and the next
starts the old grad-vector is no loner needed.
--------------------------------------------------------


2) Garbage collection
------------------------------------------
My main loop calls step each time, and I pasted your code checking the
state of the gc into my loop.

First some (selected) lines:
[3] pool:1048576 used:5072 free:19504 FREE:28 PAGE:3
[24] pool:3997696 used:10224 free:18448 FREE:81 PAGE:12
[38] pool:3997696 used:5200 free:19376 FREE:230 PAGE:10
[66] pool:16121856 used:25280 free:15680 FREE:300 PAGE:49
[77] pool:22675456 used:5200 free:19376 FREE:276 PAGE:71

So far everything is fine. Memory usage seems to fluctuate, steadily
upwards with each iteration, and dropping to ~5200 with irregular intervals.

>From here on, however, it never reaches down to the same levels again,
and (restricting to 200 iterations) the end of the output is

[199] pool:99352576 used:35744 free:29792 FREE:930 PAGE:315
[200] pool:99680256 used:38272 free:27264 FREE:862 PAGE:317

I would like it to do gc before my system starts swapping.
My interpretation of the above is that it does gc to rarely.


After I insert a fullCollect() every iteration, my output ends with

[193] pool:4653056 used:2736 free:17744 FREE:95 PAGE:14
[194] pool:5308416 used:2704 free:17776 FREE:107 PAGE:16
[195] pool:5963776 used:2736 free:17744 FREE:119 PAGE:18
[196] pool:6619136 used:2720 free:17760 FREE:131 PAGE:20
[197] pool:7274496 used:2736 free:17744 FREE:143 PAGE:22
[198] pool:7929856 used:2736 free:17744 FREE:155 PAGE:24
[199] pool:8585216 used:2736 free:17744 FREE:167 PAGE:26
[200] pool:9240576 used:2736 free:17744 FREE:179 PAGE:28
and I can see that this is much better.

My next guess would be to include calls to the generational garbage
collector at each iteration (or when memory exceeds a certain amount),
but I do not know would be the best solution.




Thomas Kuehne wrote:
> Johan Grýnqvist schrieb am 2006-03-13:
>>>I have now transcribed a program from c++ to D, but when I run it, it
>>>uses all memory and crashes.
>>>
>>>I try to minimise a function of many variables iteratively using a
>>>steepest descent method. The iteration steps are performed in a function
>>>named step.
>>>
>>>Step uses two vectors, and these are allocated anew in each step. After
>>>the step-function has finished, I expected these vectors to be
>>>garbage-collected and disappear.
>>>
>>>I allocate vectors with:
>>>  real[] grad;
>>>  grad.lenth=1000;
>>>  for (uint ix = 0; ix<grad.lenth; ix++) grad[ix] = gradient(ix);
>>>
>>>I the use the vectors but do nothing to deallocate them or indicate that
>>>I am done using them.
>>>
>>>Memory usage grows until the program need too much memory. My
>>>interpretation is that new vectors are allocated at each iteration, but
>>>the old vectors are not freed.
>>>
>>>The behaviour is the same using DMD and GDC.
> 
> Does grad.length increase with each step call? (potential fragmentation)
> Is step an anonymous function or is it nested in an other function?
> Can you replace 
> 
> real[] grad;
> grad.length = 1000;
> 
> with
> 
> real[1000] grad;
> 
> ? (depends on you problem and your design)
> 
> Neither the internal GC stats nor external tools did show any GC
> misbehaviour:
> 
> # import std.random;
> # import std.gc;
> # import std.stdio;
> # 
> # real gradient(size_t x){
> #     return x * rand();
> # }
> # 
> # void step(){
> #     real[] grad;
> #     grad.length = 1000;
> #     
> #     for(size_t ix = 0; ix < grad.length; ix++){
> #         grad[ix] = gradient(ix);
> #     }
> # }
> # 
> # int main(){
> #     size_t i = 0;
> #     GCStats gc;
> #     while(i++ < i.max){
> #         step();
> #         getStats(gc);    
> #         writefln("[%s] pool:%s used:%s free:%s FREE:%s PAGE:%s",
> #             i, gc.poolsize, gc.usedsize, gc.freelistsize,
> #             gc.freeblocks, gc.pageblocks);
> #     }
> #     return 0;
> # }
> 
> Thomas
> 
> 



More information about the Digitalmars-d-learn mailing list