D memory consumption/runtime speed problem

[bearophile]

sybrandy:
> Now, if I set the second parameter to 1, it runs decently fast.  26 
> seconds on my work laptop for a half-sized enwiki8 from the Hutter 
> challenge.  If I set it to 2, then it takes about 142 seconds.  In both 
> cases a lot of memory is used and I'm not really sure why.  Also, after 
> it prints out the results, it takes several seconds for the program to exit.

My suggestions:
- Stop using array joining, those create many small arrays that the GC has to manage, and the current D GC is not efficient at all. There are many ways to avoid array joinings and avoid most runtime allocations.
- If you can, at the end of the program you can add std.c.stdlib.exit(0);, this kills the GC mid-way in a faster way (but probably destructors are not called, so be careful).
- Use the latest daily build of the LDC compiler with very good command line arguments, and if needed link-time optimizaiton activated too :-)

And then please tell us how much time it takes to run :-)

Bye,
bearophile