Using iopipe to stream a gzipped file
Steven Schveighoffer
schveiguy at yahoo.com
Wed Jan 3 16:09:19 UTC 2018
On 1/3/18 9:45 AM, Andrew wrote:
> Hi,
>
> I have a very large gziped text file (all ASCII characters and ~500GB)
> that I want to stream and process line-by-line, and I thought the iopipe
> library would be perfect for this, but I can't seem to get it to work.
> So far, this is the closest I have to getting it to work:
>
> import iopipe.textpipe;
> import iopipe.zip;
> import iopipe.bufpipe;
> import iopipe.stream;
>
> void main()
> {
>
> auto fileToRead = openDev("file.gz").bufd.unzip(CompressionFormat.gzip);
>
> foreach (line; fileToRead.assumeText.byLineRange!false)
> {
> \\ do stuff
> }
> }
>
> but this only processes the first ~200 odd lines (I guess the initial
> read into the buffer). Can anyone help me out?
Do you have a sample file I can play with? Your iopipe chain looks
correct, so I'm not sure why it wouldn't work.
-Steve
More information about the Digitalmars-d-learn
mailing list