Read text file fast, how?

[Andrei Alexandrescu via Digitalmars-d]

On 7/25/15 8:19 AM, Johan Holmberg via Digitalmars-d wrote:
> Hi!
>
> I am trying to port a program I have written earlier to D. My previous
> versions are in C++ and Python. I was hoping that a D version would be
> similar in speed to the C++ version, rather than similar to the Python
> version. But currently it isn't.
>
> Part of the problem may be that I haven't learned the idiomatic way to
> do things in D. One such thing is perhaps: how do I read large text
> files in an efficient manner in D?
>
> Currently I have created a little test-program that does the same job as
> the UNIX-command "wc -lc", i.e. counting the number of lines and
> characters in a file. The timings I get in different languages are:
>
> D:           15s
> C++:       1.1s
> Python:   3.7s
> Perl:        2.9s

I think this harkens back to the problem discussed here:

http://stackoverflow.com/questions/28922323/improving-line-wise-i-o-operations-in-d/29153508

As I discuss there, the performance bug has been fixed for 2.068. With 
your code:

$ time wc -l <(repeat 1000000 echo hello)
  1000000 /dev/fd/11
wc -l <(repeat 1000000 echo hello)  0.11s user 2.35s system 54% cpu 
4.529 total
$ time ./test.d <(repeat 1000000 echo hello)
  1000000 6000000 /dev/fd/11
./test.d <(repeat 1000000 echo hello)  0.73s user 1.76s system 64% cpu 
3.870 total

The compilation was flag free (no -O -inline -release etc).


Andrei