Problems with GC, trees and array concatenation

[Oskar Linde]

Babele Dunnit wrote:
> Hello everybody,
> 
> I had a HELL of a day to understand what was wrong and to strip it out from my sources, but now I got it. Run this and keep an eye on the Task Manager: your memory will be eaten until death. Please note that:
> 
> 1) the keys of it all seems to be the "recursive" Individual class (I was playing with trees). Substitute the
> 
> class Individual
> {
>     Individual[20] children;
> }
> 
> with a 
> 
> class Individual
> {
>     real[20] values;
> }
> 
> and the GC will work flawlessly.
> 
> 2) please note that I am using static arrays, but dynamic arrays behave in the same way.
> 
> 3) If you use array slicing and direct assignments everything is OK. Please try both "loseMemory" and "everythingOk" versions.

I tried both with GDC 0.23 on OSX, and neither of them leaked any 
memory, so I can only speculate about what is causing this. The D GC is 
not precise (although it has become much better since 1.001). There will 
almost always be some spurious pointers. Allocating large arrays (such 
as what happens behind the scenes when doing array concatenation) will 
always carry a risk of allocated memory being hit by a spurious pointer. 
The risk of your application having runaway or contained memory leaks 
depend on the allocation patterns you have, but in general, the larger 
your objects (like arrays) are, the larger the chance that you need to 
handle their memory manually. The problem is increased if you have many 
objects containing mixed pointer and non-pointer, but apparantly random 
data.

So in conclusion, I am quite sure the following line:

              indi[] = testPop1.individuals ~ testPop2.individuals;

Is the cause of the memory leak, because it allocates a huge chunk of 
memory that is left to the GC to free. The chance of those huge chunks 
being hit by a spurious pointer seems quite high.

I have no idea why the Individual->real substitution above would make 
any difference, if not by pure chance.

/Oskar