Speeding up text file parser (BLAST tabular format)

[John Colvin via Digitalmars-d-learn]

On Tuesday, 15 September 2015 at 09:09:00 UTC, Kagamin wrote:
> On Tuesday, 15 September 2015 at 08:53:37 UTC, Fredrik Boulund 
> wrote:
>>> my favourite for streaming a file:
>>> enum chunkSize = 4096;
>>> File(fileName).byChunk(chunkSize).map!"cast(char[])a".joiner()
>>
>> Is this an efficient way of reading this type of file? What 
>> should one keep in mind when choosing chunkSize?

reasonably efficient, yes. See http://stackoverflow.com/a/237495 
for a discussion of chunk sizing when streaming a file.

> It provides you only one char at a time instead of a whole 
> line. It will be quite constraining for your code if not 
> mind-bending.

http://dlang.org/phobos/std_string.html#.lineSplitter

File(fileName).byChunk(chunkSize).map!"cast(char[])a".joiner().lineSplitter()