Help optimizing UnCompress for gzipped files

[Steven Schveighoffer]

On 1/3/18 9:42 AM, Steven Schveighoffer wrote:
> On 1/3/18 2:47 AM, Christian Köstlin wrote:
>> On 02.01.18 21:13, Steven Schveighoffer wrote:
>>> Well, you don't need to use appender for that (and doing so is copying a
>>> lot of the data an extra time). All you need is to extend the pipe until
>>> there isn't any more new data, and it will all be in the buffer.
>>>
>>> // almost the same line from your current version
>>> auto mypipe = openDev("../out/nist/2011.json.gz")
>>>                    .bufd.unzip(CompressionFormat.gzip);
>>>
>>> // This line here will work with the current release (0.0.2):
>>> while(mypipe.extend(0) != 0) {}
>> Thanks for this input, I updated the program to make use of this method
>> and compare it to the appender thing as well.
>>
> 
> Hm.. the numbers are worse! I would have expected to be at least 
> comparable. I'll have to look into it. Thanks for posting this.

Alright. I have spent some time examining the issues, and here are my 
findings:

1. The major differentiator between the C and D algorithms is the use of 
C realloc. This one thing saves the most time. I'm going to update 
iopipe so you can use it (stand by). I will also be examining how to 
simulate using realloc when not using C malloc in iopipe. I think it's 
the copying of data to the new buffer that is causing issues.

2. extend(0) always attempts to read 8k more bytes. The buffer extends 
by 8k by going into another couple pages. Somehow this is choking the 
algorithm. I think the cost of extending pages actually ends up hurting 
performance (something we need to look at in the GC in general).

3. extendElems should allow extending all the elements in the most 
optimal fashion. Fixing that, iopipe performs as well as the 
Appender/iopipe version. This is coming out as well.

Stay tuned, there will be updates to iopipe to hopefully make it as fast 
in this microbenchmark as the C version :)

-Steve