Large Arrays and GC

[dsimcha]

Acutally, if I stick a delete in my original test case at the end of test(), the
problem seems to go away.  Furthermore, the ridiculously high memory usage even
for cases of smaller arrays, goes down to like 41 megs instead of 350 for a
uint[10_000_000].  Adding a hasNoPointers seems unnecessary.  Why is the Phobos
garbage collector so bad at freeing unused dynamic arrays?

== Quote from Sivo Schilling (sivo.schilling at web.de)'s article
> This bug also occurs using DMD 1.029. It seems one have to help phobos
> gc for such cases (intended as a workaround). If you excplicitly deletes your
large array than there's no error.
> ---
> import std.stdio, std.gc;
> void main()
> {
>     char[] buf;
>     uint[] stuff;
>     uint count=0;
>     while(true && count < 100)
>     {
>         testa(stuff);
>         // fullCollect();
>         delete stuff;
>         writefln(++count);
>     }
>     readln(stdin, buf);
> }
> void testa(ref uint[] stuff)
> {
>     stuff=new uint[15_000_000];
>     // improve execution speed
>     std.gc.hasNoPointers(cast(void*)stuff);
> }
> ---
> This works with DMD 1.029 and DMD 2.013 and phobos (using Wiin XP SP3, 4 GB RAM).
> dsimcha Wrote:
> > After further testing, I've found an exact threshold for this bug.  When an array
> > of uints gets to 48_693_248 bytes (12_173_312 elements) this problem occurs, after
> > 26 iterations at the threshold, or less for larger arrays.  Anything below that,
> > even one element smaller, and memory usage is stable over at least hundreds of
> > iterations.  It appears that the number of bytes is the key, since a ulong[] will
> > allow the same number of bytes (1/2 the elements) before causing problems, and a
> > ushort[] will allow twice as many elements (same number of bytes) without
> > crashing.  Furthermore, using equivalent sized floats instead of ints (float
> > instead of uint, double instead of ulong) or using signed ints, has no effect.
> >
> > == Quote from dsimcha (dsimcha at yahoo.com)'s article
> > > Because of some difficulties encountered using D for large arrays (See previous
> > > posts about array capacity fields), I produced the following test case that
seems
> > > to be a reproducible bug in D 2.0.13.  The following program keeps allocating a
> > > huge array in a function and then letting all references to this array go out of
> > > scope.  This should result in the array being freed as soon as more memory is
> > needed.
> > > import std.stdio, std.gc;
> > > void main(){
> > >     uint count=0;
> > >     while(true) {
> > >         test();
> > >         fullCollect();
> > >         writefln(++count);
> > >     }
> > > }
> > > void test() {
> > >     uint[] stuff=new uint[15_000_000];
> > > }
> > > This produced an out of memory error after 21 iterations on my machine w/ 2
GB of
> > > RAM.  Using an array size of 10_000_000 instead of 15_000_000, its memory usage
> > > stabilized at 350 megs, which seems rather large since a uint[10_000_000] should
> > > only use 40 megs, plus maybe another 40 for overhead.  Furthermore, it takes
> > > several iterations for the memory usage to reach this level.  Using a larger
array
> > > size, such as 100_000_000, made this test run out of memory after even fewer
> > > iterations.  Furthermore, changing from a uint[] to real[] with a size of
> > > 15_000_000 made it run out after 8 iterations instead of 21.
> >