Using iopipe to stream a gzipped file
    Andrew 
    aabrown24 at hotmail.com
       
    Wed Jan  3 14:45:26 UTC 2018
    
    
  
Hi,
I have a very large gziped text file (all ASCII characters and 
~500GB) that I want to stream and process line-by-line, and I 
thought the iopipe library would be perfect for this, but I can't 
seem to get it to work. So far, this is the closest I have to 
getting it to work:
import iopipe.textpipe;
import iopipe.zip;
import iopipe.bufpipe;
import iopipe.stream;
void main()
{
   auto fileToRead = 
openDev("file.gz").bufd.unzip(CompressionFormat.gzip);
   foreach (line; fileToRead.assumeText.byLineRange!false)
   {
      \\ do stuff
   }
}
but this only processes the first ~200 odd lines (I guess the 
initial read into the buffer). Can anyone help me out?
Thanks very much
Andrew
    
    
More information about the Digitalmars-d-learn
mailing list