Speed of csvReader

[Chris Wright via Digitalmars-d-learn]

On Tue, 26 Jan 2016 18:16:28 +0000, Gerald Jansen wrote:

> On Thursday, 21 January 2016 at 21:24:49 UTC, H. S. Teoh wrote:
>>
>> While this is no fancy range-based code, and one might say it's more
>> hackish and C-like than idiomatic D, the problem is that current D
>> compilers can't quite optimize range-based code to this extent yet.
>> Perhaps in the future optimizers will improve so that more idiomiatic,
>> range-based code will have comparable performance with fastcsv. (At
>> least in theory this should be possible.)
> 
> As a D novice still struggling with the concept that composable
> range-based functions can be more efficient than good-old looping (ya, I
> know, cache friendliness and GC avoidance), I find it extremely
> interesting that someone as expert as yourself would reach for a C-like
> approach for serious data crunching. Given that data crunching is the
> kind of thing I need to do a lot, I'm wondering how general your
> statement above might be at this time w.r.t. this and possibly other
> domains.

You want to reduce allocations. Ranges often let you do that. However, 
it's sometimes unsafe to reuse range values that aren't immutable. That 
means, if you want to keep the values around, you need to copy them -- 
which introduces an allocation.

You can get fewer large allocations by reading the whole file at once 
manually and using slices into that large allocation.