Reading bigger file

[Marco Leise]

Am Fri, 08 Mar 2013 16:31:45 +0100
schrieb "Chris Cain" <clcain at uncg.edu>:

> On Friday, 8 March 2013 at 15:25:02 UTC, bioinfornatics wrote:
> > why when reading a huge file more i advance into the file more 
> > that take time to get a line ?
> 
> ----
>      StopWatch   sw;
>      while( !fastq1.empty ){
>          sw.start();
>          auto        q1          = fastq1.next();
>          sw.stop();
>          writeln( sw.peek().msecs() );
>      }
> ----
> 
> That's because you never reset the StopWatch.

Ha ha!

On a different note... if you still just parse linearly
without seeking inside the file, you should consider parsing
the compressed file directly. GZIP decompression is very fast.
You may get 80 MiB/s for the decompression as well as for HDD
read speed. So as long as you parallelize reading new data and
decompression, that's your actual read speed. Now the
compression factor you indicated is ~15x, so that makes it
effectively a 15 * 80 MiB/s = 1.2 GiB/s read speed. Sounds
good? :)

-- 
Marco