A program leaking memory.
Johan Grönqvist
johan.gronqvist at gmail.com
Mon Mar 13 11:43:10 PST 2006
Thomas: Thak you very much for your answer!
Now I have found the cause to this problem,
The garbage collector collects so rarely that my system starts swapping
intensely.
This time I only used gdc.
(gcc version 4.0.1, gdc 0.17, using dmd 0.140)
Any hints to the solution are most welcome!
Is there a way to tell the gc never to use more than (say) 100MB of memory?
I could insert an if-clause in each step, calling the collector if some
limit is exceeded, but if feels a bit inelegant.
Comments to Thomas Kuehnes very helpful answer below.
/ johan
1) Static vs. Dynamic arrays:
--------------------------------------------------------
The system size I use will not be known at compile time, and my
understanding is that I can therefore not use
real[myLength] grad;
but have to use
real[] grad;
grad.length = myLength;
where myLength is computed earlier.
The arrays I initialize in this way are never again resized. With this I
mean that the two first things I do are
real[] grad;
grad.lenth = myLength;
thereafter I modify the elements and read off their values a lot, but I
never resize the array again, i.e., the memory need for grad should
never change during a step, and as soon as one step ends and the next
starts the old grad-vector is no loner needed.
--------------------------------------------------------
2) Garbage collection
------------------------------------------
My main loop calls step each time, and I pasted your code checking the
state of the gc into my loop.
First some (selected) lines:
[3] pool:1048576 used:5072 free:19504 FREE:28 PAGE:3
[24] pool:3997696 used:10224 free:18448 FREE:81 PAGE:12
[38] pool:3997696 used:5200 free:19376 FREE:230 PAGE:10
[66] pool:16121856 used:25280 free:15680 FREE:300 PAGE:49
[77] pool:22675456 used:5200 free:19376 FREE:276 PAGE:71
So far everything is fine. Memory usage seems to fluctuate, steadily
upwards with each iteration, and dropping to ~5200 with irregular intervals.
>From here on, however, it never reaches down to the same levels again,
and (restricting to 200 iterations) the end of the output is
[199] pool:99352576 used:35744 free:29792 FREE:930 PAGE:315
[200] pool:99680256 used:38272 free:27264 FREE:862 PAGE:317
I would like it to do gc before my system starts swapping.
My interpretation of the above is that it does gc to rarely.
After I insert a fullCollect() every iteration, my output ends with
[193] pool:4653056 used:2736 free:17744 FREE:95 PAGE:14
[194] pool:5308416 used:2704 free:17776 FREE:107 PAGE:16
[195] pool:5963776 used:2736 free:17744 FREE:119 PAGE:18
[196] pool:6619136 used:2720 free:17760 FREE:131 PAGE:20
[197] pool:7274496 used:2736 free:17744 FREE:143 PAGE:22
[198] pool:7929856 used:2736 free:17744 FREE:155 PAGE:24
[199] pool:8585216 used:2736 free:17744 FREE:167 PAGE:26
[200] pool:9240576 used:2736 free:17744 FREE:179 PAGE:28
and I can see that this is much better.
My next guess would be to include calls to the generational garbage
collector at each iteration (or when memory exceeds a certain amount),
but I do not know would be the best solution.
Thomas Kuehne wrote:
> Johan Grýnqvist schrieb am 2006-03-13:
>>>I have now transcribed a program from c++ to D, but when I run it, it
>>>uses all memory and crashes.
>>>
>>>I try to minimise a function of many variables iteratively using a
>>>steepest descent method. The iteration steps are performed in a function
>>>named step.
>>>
>>>Step uses two vectors, and these are allocated anew in each step. After
>>>the step-function has finished, I expected these vectors to be
>>>garbage-collected and disappear.
>>>
>>>I allocate vectors with:
>>> real[] grad;
>>> grad.lenth=1000;
>>> for (uint ix = 0; ix<grad.lenth; ix++) grad[ix] = gradient(ix);
>>>
>>>I the use the vectors but do nothing to deallocate them or indicate that
>>>I am done using them.
>>>
>>>Memory usage grows until the program need too much memory. My
>>>interpretation is that new vectors are allocated at each iteration, but
>>>the old vectors are not freed.
>>>
>>>The behaviour is the same using DMD and GDC.
>
> Does grad.length increase with each step call? (potential fragmentation)
> Is step an anonymous function or is it nested in an other function?
> Can you replace
>
> real[] grad;
> grad.length = 1000;
>
> with
>
> real[1000] grad;
>
> ? (depends on you problem and your design)
>
> Neither the internal GC stats nor external tools did show any GC
> misbehaviour:
>
> # import std.random;
> # import std.gc;
> # import std.stdio;
> #
> # real gradient(size_t x){
> # return x * rand();
> # }
> #
> # void step(){
> # real[] grad;
> # grad.length = 1000;
> #
> # for(size_t ix = 0; ix < grad.length; ix++){
> # grad[ix] = gradient(ix);
> # }
> # }
> #
> # int main(){
> # size_t i = 0;
> # GCStats gc;
> # while(i++ < i.max){
> # step();
> # getStats(gc);
> # writefln("[%s] pool:%s used:%s free:%s FREE:%s PAGE:%s",
> # i, gc.poolsize, gc.usedsize, gc.freelistsize,
> # gc.freeblocks, gc.pageblocks);
> # }
> # return 0;
> # }
>
> Thomas
>
>
More information about the Digitalmars-d-learn
mailing list