Using iopipe to stream a gzipped file

Steven Schveighoffer schveiguy at yahoo.com
Thu Jan 4 15:48:21 UTC 2018


On 1/4/18 7:23 AM, Andrew wrote:
> On Thursday, 4 January 2018 at 12:15:27 UTC, Steven Schveighoffer wrote:
>>
>> In any case, I'll figure out how to deal with concatenated gzip file, 
>> and update iopipe. Next version will focus on a bunch of stuff 
>> relating to the 2 zip threads recently posted here.
>>
> 
> That would be really great for me, thank you! 

It's now been updated, see version 0.0.3.

Note, the performance isn't something I focused on. I'll note that gzcat 
| wc -l is 2x faster than your simple example on that file.

I can think of a couple reasons for this:

1. gzcat may use mmap to increase read speed
2. gzcat may read larger chunks at once (this can be tuned using iopipe 
as well, just set the optimizedReadSize).
3. gzcat file.gz | iopipe_byline -nooutput is about 20% faster than 
using wc -l, so it's definitely not the line parsing.

Let me know if this works correctly for your other test cases! If not, 
file an issue: https://github.com/schveiguy/iopipe/issues

-Steve


More information about the Digitalmars-d-learn mailing list