Large Arrays and GC

[Sivo Schilling]

This bug also occurs using DMD 1.029. It seems one have to help phobos
gc for such cases (intended as a workaround). If you excplicitly deletes your large array than there's no error.
---
import std.stdio, std.gc;

void main()
{
    char[] buf;
    uint[] stuff;

    uint count=0;
    while(true && count < 100)
    {
        testa(stuff);
        // fullCollect();
        delete stuff;
        writefln(++count);
    }
    readln(stdin, buf);
}

void testa(ref uint[] stuff)
{
    stuff=new uint[15_000_000];
    // improve execution speed
    std.gc.hasNoPointers(cast(void*)stuff);
}
---
This works with DMD 1.029 and DMD 2.013 and phobos (using Wiin XP SP3, 4 GB RAM).
 
dsimcha Wrote:

> After further testing, I've found an exact threshold for this bug.  When an array
> of uints gets to 48_693_248 bytes (12_173_312 elements) this problem occurs, after
> 26 iterations at the threshold, or less for larger arrays.  Anything below that,
> even one element smaller, and memory usage is stable over at least hundreds of
> iterations.  It appears that the number of bytes is the key, since a ulong[] will
> allow the same number of bytes (1/2 the elements) before causing problems, and a
> ushort[] will allow twice as many elements (same number of bytes) without
> crashing.  Furthermore, using equivalent sized floats instead of ints (float
> instead of uint, double instead of ulong) or using signed ints, has no effect.
> 
> == Quote from dsimcha (dsimcha at yahoo.com)'s article
> > Because of some difficulties encountered using D for large arrays (See previous
> > posts about array capacity fields), I produced the following test case that seems
> > to be a reproducible bug in D 2.0.13.  The following program keeps allocating a
> > huge array in a function and then letting all references to this array go out of
> > scope.  This should result in the array being freed as soon as more memory is
> needed.
> > import std.stdio, std.gc;
> > void main(){
> >     uint count=0;
> >     while(true) {
> >         test();
> >         fullCollect();
> >         writefln(++count);
> >     }
> > }
> > void test() {
> >     uint[] stuff=new uint[15_000_000];
> > }
> > This produced an out of memory error after 21 iterations on my machine w/ 2 GB of
> > RAM.  Using an array size of 10_000_000 instead of 15_000_000, its memory usage
> > stabilized at 350 megs, which seems rather large since a uint[10_000_000] should
> > only use 40 megs, plus maybe another 40 for overhead.  Furthermore, it takes
> > several iterations for the memory usage to reach this level.  Using a larger array
> > size, such as 100_000_000, made this test run out of memory after even fewer
> > iterations.  Furthermore, changing from a uint[] to real[] with a size of
> > 15_000_000 made it run out after 8 iterations instead of 21.
>