Regarding hex strings

[Marco Leise]

Am Thu, 18 Oct 2012 16:31:57 +0200
schrieb "monarch_dodra" <monarchdodra at gmail.com>:

> On Thursday, 18 October 2012 at 13:15:55 UTC, bearophile wrote:
> > monarch_dodra:
> >
> >> hex! was a very good idea actually, imo.
> >
> > It must scale up to "real world" usages. Try it with a program
> > composed of 3 modules each one containing a 100 KB long string.
> > Then try it with a program with two hundred of medium sized
> > literals, and let's see compilation times and binary sizes.
> >
> > Bye,
> > bearophile
> 
> Hum... The compilation is pretty fast actually, about 1 second, 
> provided it doesn't choke.
> 
> It works for strings up to a length of 400 lines @ 80 chars per 
> line, which result to approximately 16K of data. After that, I 
> get a DMD out of memory error.
> 
> DMD memory usage spikes quite quickly. To compile those 400 lines 
> (16K), I use 800MB of memory (!). If I reach about 1GB, then it 
> crashes.
> 
> I tried using a refAppender instead of ret~, but that changed 
> nothing.
> 
> Kind of weird it would use that much memory though...
> 
> Also, the memory doesn't get released. I can parse a 1x400 Line 
> string, but if I try to parse 3 of them, DMD will choke on the 
> second one. :(

Hehe, I assume most of the regulars know this: DMD used to
use a garbage collector that is disabled. Memory just isn't
freed! Also it has copy on write semantics during CTFE:

int bug6498(int x)
{
    int n = 0;
    while (n < x)
        ++n;
    return n;
}
static assert(bug6498(10_000_000)==10_000_000);

--> Fails with an 'out of memory' error.

http://d.puremagic.com/issues/show_bug.cgi?id=6498

So, as strange as it sounds, for now try not to write often or
into large blocks. Using this knowledge I was sometimes able
to bring down the memory consumption considerably by caching
recurring concatenations of two strings or to!string calls.

That said, appending single elements to an array may actually
be better than using a fixed-sized one and have DMD duplicate
it on every write. :p

Please remember to give Don a cookie when he manages to change
the compiler to modify in-place where appropriate.

-- 
Marco