Running out of memory

FG home at fgda.pl
Thu Dec 27 04:30:06 PST 2012


Hello,
I'm having memory problems with a string concatenation program.
I'm on Windows 7 64-bit with DMD32 v2.060 and cygwin gcc-3.4.4 32-bit.
Consider this example in C++:

     #include <iostream>
     int main(int argc, char **args) {
         std::string s;
         for (int i=0, j=0; i < 500000000; i++) {
             s += "....";
             if (i % 10000000 == 0) {
                 std::cout << "loop " << ++j << std::endl;
                 s = "";
             }
         }
         return 0;
     }

After the first "loop" (10 million iterations) the memory footprint is stable at 
40 MB, which is expected. Now, rewriting that to D:

     import std.stdio;
     class C { string x; }
     void main(string[] args) {
         char[] s;
         // s.reserve(40_000_000);
         for (int i=0, j=0; i < 500_000_000; i++) {
             s ~= "....";
             if (i % 10_000_000 == 0) {
                 writeln("loop ", ++j); stdout.flush();
                 s = "".dup;
             }
         }
     }

It runs 6 times faster thanks to a better string implementation than C++ (hats 
off), but memory keeps growing long after the first "loop" message, reaching 
around 350-430 MB, then sometimes near the end it starts growing again, reaching 
even 700 MB. Why doesn't the GC take care of it? Interestingly, when I uncomment 
the s.reserve(...) line, it's much worse - memory keeps growing until the 
program dies with a core.exception.OutOfMemoryError.

Then I have rewritten this code to use an Appender:

     import std.stdio, std.array;
     class C { string x; }
     void main(string[] args) {
         char[] s;
         s.reserve(40_000_000);
         auto app = appender(s);
         for (int i=0, j=0; i < 500_000_000; i++) {
             app.put("....");
             if (i % 10_000_000 == 0) {
                 writeln("loop ", ++j); stdout.flush();
                 app.clear();
             }
         }
     }

This is what I was after - it runs 4 times faster than the first D code and uses 
only 40 MB of memory. It's close to the efficiency of operating on the array 
using indexing. I get, that it's by design much faster than s ~= "....", but why 
is so much memory wasted in the first D example?
Am I doing something wrong? I thought s ~= a is rewritten as an equivalent of 
s.push_back(a) (from C++'s vector) and not s = s ~ a.

best regards,
FG


More information about the Digitalmars-d-learn mailing list