Large Arrays and GC

dsimcha dsimcha at yahoo.com
Thu May 8 16:54:51 PDT 2008


Another interesting point:  This bug also occurs on GDC 0.24, but here the
threshold is lower, 33_423_360 bytes.  Must be something pretty deep.


== Quote from Sean Kelly (sean at invisibleduck.org)'s article
> Well whatever's going on, it's not tracked by gc_stats().  I modified
> the previous functions slightly to ensure that both allocated exactly
> the same amount of memory, then verified this via a printf inside the GC
> code--malloc calls were for 60000000 bytes:
> void test() {
>      void* stuff = (new byte[15_000_000 * 4 - 1]).ptr;
> }
> void test() {
>      void* stuff=GC.malloc(15_000_000 * 4, GC.BlkAttr.NO_SCAN);
> }
> Output using gc_stats() on each iteration after the collect was
> identical for both function calls after things had stabalized:
> *** stats ***
> poolsize: 360251392
> usedsize: 640
> freeblocks: 56
> freelistsize: 7552
> pageblocks: 6
> What I don't understand is why Task Manager is reporting the GC.malloc
> version using only 18 megs while the other uses 370, since everything
> I've looked at internally so far suggests they should be the same.
> Perhaps gc_stats() isn't reporting something properly--I really haven't
> given that routine a very close look.
> Sean Kelly wrote:
> > I'm not sure what to say.  This sample works fine with D 1.0 using
> > Tango, though the memory usage is strangely high at around 370 megs.
> > Here's the weird thing, I tried running these two versions of the test
> > function for comparison:
> >
> > void test() {
> >     void* stuff = (new byte[15_000_000 * 4]).ptr;
> > }
> >
> > void test() {
> >     void* stuff=GC.malloc(15_000_000 * 4, GC.BlkAttr.NO_SCAN);
> > }
> >
> > The first one uses a stable 370 megs of memory and the second a stable
> > 18 megs.  Obviously there's something weird going on with how arrays are
> > handled.
> >
> > dsimcha wrote:
> >> After further testing, I've found an exact threshold for this bug.
> >> When an array
> >> of uints gets to 48_693_248 bytes (12_173_312 elements) this problem
> >> occurs, after
> >> 26 iterations at the threshold, or less for larger arrays.  Anything
> >> below that,
> >> even one element smaller, and memory usage is stable over at least
> >> hundreds of
> >> iterations.  It appears that the number of bytes is the key, since a
> >> ulong[] will
> >> allow the same number of bytes (1/2 the elements) before causing
> >> problems, and a
> >> ushort[] will allow twice as many elements (same number of bytes) without
> >> crashing.  Furthermore, using equivalent sized floats instead of ints
> >> (float
> >> instead of uint, double instead of ulong) or using signed ints, has no
> >> effect.
> >>
> >> == Quote from dsimcha (dsimcha at yahoo.com)'s article
> >>> Because of some difficulties encountered using D for large arrays
> >>> (See previous
> >>> posts about array capacity fields), I produced the following test
> >>> case that seems
> >>> to be a reproducible bug in D 2.0.13.  The following program keeps
> >>> allocating a
> >>> huge array in a function and then letting all references to this
> >>> array go out of
> >>> scope.  This should result in the array being freed as soon as more
> >>> memory is
> >> needed.
> >>> import std.stdio, std.gc;
> >>> void main(){
> >>>     uint count=0;
> >>>     while(true) {
> >>>         test();
> >>>         fullCollect();
> >>>         writefln(++count);
> >>>     }
> >>> }
> >>> void test() {
> >>>     uint[] stuff=new uint[15_000_000];
> >>> }
> >>> This produced an out of memory error after 21 iterations on my
> >>> machine w/ 2 GB of
> >>> RAM.  Using an array size of 10_000_000 instead of 15_000_000, its
> >>> memory usage
> >>> stabilized at 350 megs, which seems rather large since a
> >>> uint[10_000_000] should
> >>> only use 40 megs, plus maybe another 40 for overhead.  Furthermore,
> >>> it takes
> >>> several iterations for the memory usage to reach this level.  Using a
> >>> larger array
> >>> size, such as 100_000_000, made this test run out of memory after
> >>> even fewer
> >>> iterations.  Furthermore, changing from a uint[] to real[] with a
> >>> size of
> >>> 15_000_000 made it run out after 8 iterations instead of 21.
> >>




More information about the Digitalmars-d mailing list