[Issue 1382] New: memory allocated for arrays in CTFE functions during compilation is not released

[d-bugmail at puremagic.com]

http://d.puremagic.com/issues/show_bug.cgi?id=1382

           Summary: memory allocated for arrays in CTFE functions during
                    compilation is not released
           Product: D
           Version: 1.019
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: DMD
        AssignedTo: bugzilla at digitalmars.com
        ReportedBy: kamm-removethis at incasoftware.de


This problem is encountered when large arrays are manipulated during CTFE.
Symtoms are dmd allocating a lot of memory, eating up all swap and finally
terminating with an out of memory error.

The core of the problem can be seen in this code
--
char[] make_empty_string(int n)
{
  char[] result;

  for(int i = 0; i < n; ++i)
    result ~= " ";
  return result;
}

const char[] largestring = make_empty_string(100000);
void main() {}
---

This snippet will require 5 gb of memory to compile instead of the mere 100 kb
the final string will require. It is caused by the intermediate strings stored
in result during the iteration never being discarded. The problem is not
limited to concatenation, modifications of a single element of the array will
also cause the whole array to be duplicated. 

While this particular piece of code can be rewritten to consume less memory,
that's not generally possible. An example where reduction is not possible is
splitting a string into substrings.

It does come up in practice: Someone wanted to generate a function to get the
Unicode general catergory for a dchar from the textfiles from the Unicode
Character Database and ran into this issue. I wanted to parse the D BNF to
generate a parser at compile time and had dmd exit with an out of memory error.

It seems CTFE needs a compile time garbage collector.


--