Need a Faster Compressor

[Era Scarecrow via Digitalmars-d]

On Sunday, 22 May 2016 at 16:06:07 UTC, Marco Leise wrote:
> There are a few open questions, for me at least. Are there 
> other proposals with good properties? Are there any objections 
> against the benchmarking method or the license? Can the 
> implementation be in D? If not, should we link against the 
> system liblz4.so/a or copy the C code into the backend? Is a 
> pre-filled dictionary worthwhile? If so, which words should go 
> in it? The latter is important because changes to the Dlang 
> mangling need to be reflected in other projects outside our 
> direct control, too.

How it's being benchmarked does make me question it a bit. I 
avoided doing multi-threading so my results are as they would run 
on 1 CPU/Core, while adding multi-threading probably isn't too 
hard to do (at least in D). Then there's the wonder of how many 
cores, and other details. A compressor designed to run in 
parallel will naturally be in it's own element there. This of 
course assumes we are allowed to in the first place; The compiler 
may already be using those cores for other tasks: Scanning other 
files, optimizing code, CTFE, etc, to which point speed gained in 
compression may not actually translate to an overall speedup if 
you are stealing CPU cycles.

  As for a dictionary and being primed, I'm trying to figure out 
what would be in it myself. Commonly used templates and keywords 
that appear in symbols are so far my best guess. I'd avoid 
library names unless they'd appear in the symbols (so 'std.core' 
might not mean much, while 'unittestL' does). Then again I don't 
have a good sample of what I'd actually encounter live, maybe 
every major library import would be good thing to have.