Help optimizing UnCompress for gzipped files

[Christian Köstlin]

On 02.01.18 21:13, Steven Schveighoffer wrote:
> Well, you don't need to use appender for that (and doing so is copying a
> lot of the data an extra time). All you need is to extend the pipe until
> there isn't any more new data, and it will all be in the buffer.
> 
> // almost the same line from your current version
> auto mypipe = openDev("../out/nist/2011.json.gz")
>                   .bufd.unzip(CompressionFormat.gzip);
> 
> // This line here will work with the current release (0.0.2):
> while(mypipe.extend(0) != 0) {}
Thanks for this input, I updated the program to make use of this method
and compare it to the appender thing as well.

>> I will give the direct gunzip calls a try ...
I added direct gunzip calls as well... Those are really good, as long as
I do not try to get the data into ram :) then it is "bad" again.
I wonder what the real difference to the lowlevel solution with own
appender and the c version is. For me they look almost the same (ugly,
only the performance seems to be nice).

Funny thing is, that if I add the clang address sanitizer things to the
c program, I get almost the same numbers as for java :)


> Yeah, with jsoniopipe being very raw, I wouldn't be sure it was usable
> in your case. The end goal is to have something fast, but very easy to
> construct. I wasn't planning on focusing on the speed (yet) like other
> libraries do, but ease of writing code to use it.
> 
> -Steve

--
Christian Köstlin