Using iopipe to stream a gzipped file
Steven Schveighoffer
schveiguy at yahoo.com
Thu Jan 4 15:48:21 UTC 2018
On 1/4/18 7:23 AM, Andrew wrote:
> On Thursday, 4 January 2018 at 12:15:27 UTC, Steven Schveighoffer wrote:
>>
>> In any case, I'll figure out how to deal with concatenated gzip file,
>> and update iopipe. Next version will focus on a bunch of stuff
>> relating to the 2 zip threads recently posted here.
>>
>
> That would be really great for me, thank you!
It's now been updated, see version 0.0.3.
Note, the performance isn't something I focused on. I'll note that gzcat
| wc -l is 2x faster than your simple example on that file.
I can think of a couple reasons for this:
1. gzcat may use mmap to increase read speed
2. gzcat may read larger chunks at once (this can be tuned using iopipe
as well, just set the optimizedReadSize).
3. gzcat file.gz | iopipe_byline -nooutput is about 20% faster than
using wc -l, so it's definitely not the line parsing.
Let me know if this works correctly for your other test cases! If not,
file an issue: https://github.com/schveiguy/iopipe/issues
-Steve
More information about the Digitalmars-d-learn
mailing list