Help optimizing UnCompress for gzipped files

[Steven Schveighoffer]

On 1/2/18 1:01 PM, Christian Köstlin wrote:
> On 02.01.18 15:09, Steven Schveighoffer wrote:
>> On 1/2/18 8:57 AM, Adam D. Ruppe wrote:
>>> On Tuesday, 2 January 2018 at 11:22:06 UTC, Stefan Koch wrote:
>>>> You can make it much faster by using a sliced static array as buffer.
>>>
>>> Only if you want data corruption! It keeps a copy of your pointer
>>> internally: https://github.com/dlang/phobos/blob/master/std/zlib.d#L605
>>>
>>> It also will always overallocate new buffers on each call
>>> <https://github.com/dlang/phobos/blob/master/std/zlib.d#L602>
>>>
>>> There is no efficient way to use it. The implementation is substandard
>>> because the API limits the design.
>>
>> iopipe handles this quite well. And deals with the buffers properly
>> (yes, it is very tricky. You have to ref-count the zstream structure,
>> because it keeps internal pointers to *itself* as well!). And no, iopipe
>> doesn't use std.zlib, I use the etc.zlib functions (but I poached some
>> ideas from std.zlib when writing it).
>>
>> https://github.com/schveiguy/iopipe/blob/master/source/iopipe/zip.d
>>
>> I even wrote a json parser for iopipe. But it's far from complete. And
>> probably needs updating since I changed some of the iopipe API.
>>
>> https://github.com/schveiguy/jsoniopipe
>>
>> Depending on the use case, it might be enough, and should be very fast.
>>
> Thanks Steve for this proposal (actually I already had an iopipe version
> on my harddisk that I applied to this problem) Its more or less your
> unzip example + putting the data to an appender (I hope this is how it
> should be done, to get the data to RAM).

Well, you don't need to use appender for that (and doing so is copying a 
lot of the data an extra time). All you need is to extend the pipe until 
there isn't any more new data, and it will all be in the buffer.

// almost the same line from your current version
auto mypipe = openDev("../out/nist/2011.json.gz")
                   .bufd.unzip(CompressionFormat.gzip);

// This line here will work with the current release (0.0.2):
while(mypipe.extend(0) != 0) {}

//But I have a fix for a bug that hasn't been released yet, this would 
work if you use iopipe-master:
mypipe.ensureElems();

// getting the data is as simple as looking at the buffer.
auto data = mypipe.window; // ubyte[] of the data

> iopipe is already better than the normal dlang version, almost like
> java, but still far from the solution. I updated
> https://github.com/gizmomogwai/benchmarks/tree/master/gunzip
> 
> I will give the direct gunzip calls a try ...
> 
> In terms of json parsing, I had really nice results with the fast.json
> pull parser, but its comparing a little bit apples with oranges, because
> I did not pull out all the data there.

Yeah, with jsoniopipe being very raw, I wouldn't be sure it was usable 
in your case. The end goal is to have something fast, but very easy to 
construct. I wasn't planning on focusing on the speed (yet) like other 
libraries do, but ease of writing code to use it.

-Steve