Need a Faster Compressor

[Marco Leise via Digitalmars-d]

Am Sun, 22 May 2016 14:06:52 +0000
schrieb Stefan Koch <uplink.coder at googlemail.com>:

> On Sunday, 22 May 2016 at 14:05:23 UTC, Marco Leise wrote:
> > ⬅ Download proof of concept
> >
> > This is a proof of concept micro-benchmark compressing the 
> > 37_640 bytes symbol from the bug report linked above with both 
> > id_compress from the dmd backend and lz4 (without dictionary). 
> > The lz4 result is additionally transformed to 7-bits similar to 
> > base-64.
> >
> > Original size: 37640 bytes
> >
> > id_compress : 5243 ms for 10_000 iterations, 800 bytes
> > lz4         :   71 ms for 10_000 iterations, 389 bytes
> >
> > That roughly corresponds to a 2x higher compression ratio at 
> > 70x faster compression speed.  
> 
> Please submit a PR.

There are a few open questions, for me at least. Are there
other proposals with good properties? Are there any objections
against the benchmarking method or the license? Can the
implementation be in D? If not, should we link against the
system liblz4.so/a or copy the C code into the backend? Is a
pre-filled dictionary worthwhile? If so, which words should go
in it? The latter is important because changes to the Dlang
mangling need to be reflected in other projects outside our
direct control, too.

-- 
Marco