[Issue 24328] New: Very poor GC memory utilization due to fragmentation

[d-bugmail at puremagic.com]

https://issues.dlang.org/show_bug.cgi?id=24328

          Issue ID: 24328
           Summary: Very poor GC memory utilization due to fragmentation
           Product: D
           Version: D2
          Hardware: x86_64
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P1
         Component: druntime
          Assignee: nobody at puremagic.com
          Reporter: dlang-bugzilla at thecybershadow.net

This program seems to be able to utilize only 2% of memory available to it -
the rest seems to be wasted by fragmentation:

///////////////////////////////// test.d ////////////////////////////////
import core.stdc.stdio;
import core.stdc.stdlib;

enum size_t memoryLimit = 1UL * 1024 * 1024 * 1024;
enum size_t allocationSize = 64;

void main()
{
    {
        import core.sys.posix.sys.resource :
            rlimit, RLIMIT_AS, getrlimit, setrlimit;
        rlimit lim;
        getrlimit(RLIMIT_AS, &lim);
        lim.rlim_cur = memoryLimit;
        setrlimit(RLIMIT_AS, &lim);
    }

    // Size of reachable data created by this function
    size_t utilizedSize;

    scope(exit) printf("Utilized size: %zu (%zd%%)\n",
        utilizedSize, utilizedSize * 100 / memoryLimit);

    ubyte[][] pinned;
    int n;
    while (true)
    {
        ubyte[] arr;
        {
            scope(failure) printf("Failed to allocate %zu\n",
                allocationSize);
            arr = new ubyte[allocationSize];
        }
        bool keep = n++ % 2 == 0;
        if (keep)
        {
            {
                scope(failure) printf("Failed to reallocate %zu bytes\n",
                    (pinned.length + 1) * pinned[0].sizeof);
                pinned ~= arr;
            }
            utilizedSize += allocationSize + pinned[0].sizeof;
        }
    }
}
/////////////////////////////////////////////////////////////////////////

--