Need a Faster Compressor

[Era Scarecrow via Digitalmars-d]

On Sunday, 22 May 2016 at 19:21:13 UTC, Walter Bright wrote:
> On 5/22/2016 3:32 AM, Era Scarecrow wrote:
>> Curiously the extra 95 symbols actually is just enough to keep 
>> compression up and bring speed to about 37% speed gain, with 
>> it looks like no disadvantages.
>
> You're doing the gods' work, Era!

  Maybe... I have to wonder if LZ4 from the looks of things will 
take over instead. Have to see how it performs (and if we can 
integrate it); Meanwhile the main advantage of id_compress (as 
I'm tinkering with it) is it makes no changes to the 
signature/behavior so taking advantage of it's boost in speed can 
be immediate (although it will no doubt have a bit weaker 
compression, and demangling needs to be re-written for it).

  Honestly the majority of speed gains with id_compress are simply 
from having a much smaller window(1023 to 220), also resulting in 
a smaller footprint for the compressed output (3 bytes to 2 
bytes).

  Compare to the speed gains from my new algorithm (at 20% faster) 
is pretty constant, and retains a HUGE window (pre-scans 8k, has 
a 2k window) that it can find matches from, so it will win in 
compression.

  If LZ4 is half as good as the initial results are then I'd go 
with it instead.