Performant method for reading huge text files

[Chris Williams]

On Tuesday, 4 February 2014 at 00:04:23 UTC, Rene Zwanenburg 
wrote:
> On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote:
>> Rene Zwanenburg:
>>
>>> The problem is speed. I'm using LockingTextReader in 
>>> std.stdio, but it't not nearly fast enough. On my system it 
>>> only reads about 3 MB/s with one core spending all it's time 
>>> in IO calls.
>>
>> Are you reading the text by lines? In Bugzilla there is a 
>> byLineFast:
>> https://d.puremagic.com/issues/show_bug.cgi?id=11810
>>
>> Bye,
>> bearophile
>
> Nope, I'm feeding it to csvReader which uses an input range of 
> characters. Come to think of it..
>
> Well this is embarassing, I've been sloppy with my profiling 
> :). It appears the time is actually spent converting strings to 
> doubles, done by csvReader to read a row into my Record struct. 
> No way to speed that up I suppose. Still I find it surprising 
> that parsing doubles is so slow.

Parsing should be faster than I/O. Set up two buffers and have 
one thread reading into buffer A while you parse buffer B with a 
second thread.