[Issue 3763] New: std.stdio.readlnImpl absurdly inefficient

[d-bugmail at puremagic.com]

http://d.puremagic.com/issues/show_bug.cgi?id=3763

           Summary: std.stdio.readlnImpl absurdly inefficient
           Product: D
           Version: 2.040
          Platform: Other
        OS/Version: Windows
            Status: NEW
          Keywords: performance
          Severity: normal
          Priority: P2
         Component: Phobos
        AssignedTo: nobody at puremagic.com
        ReportedBy: dsimcha at yahoo.com


--- Comment #0 from David Simcha <dsimcha at yahoo.com> 2010-02-01 16:18:57 PST ---
Apparently stdio.readln() copies the data it reads a ridiculous number of times
and performs a ridiculous amount of heap allocations.  The following program
uses **gigabytes** of RAM.  This issue came to my attention while reading in a
huge file in the presence of large associative arrays that contained lots of
false pointers.

import std.stdio, core.memory;

void main() {
    // Write 512 kilobytes out to a file on a single line.
    auto writeHandle = File("foo.txt", "wb");
    auto contents = new char[512 * 1024];
    contents[] = 'a';
    writeHandle.writeln(contents);
    writeHandle.close;
    contents = null;
    GC.collect();

    // Read it back with the GC disabled.
    GC.disable;
    auto readHandle = File("foo.txt");
    auto firstLine = readHandle.readln();

    stderr.writeln("Check task manager for memory usage, then press enter.");
    stdin.readln();
}

Under a sane allocation scheme (geometric growth of the buffer), this program
would only use about 1 MB of RAM plus overhead for stack space, etc., even with
the GC disabled.  (Derivation:  512 * 1024 == 2^19.  2^0 + 2^1 + ... + 2^19 ==
2^20 - 1.)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------