Phobos usability with text files

[Andrei Alexandrescu]

On 12/26/10 10:12 AM, bearophile wrote:
> This is related to this (closed) issue, but this time I prefer to discuss the topic on the newsgroup:
> http://d.puremagic.com/issues/show_bug.cgi?id=4474
>
> To show why this enhancement request is useful I use a little scripting task. A D2 program has to read a file that contains one word in each line, and print all the words with the highest length.
>
> A first naive functional style implementation (if you keep in mind that File opens files in binary mode on default, this default is different from the Python one, I am not sure what's the best default. I think most files I open are text ones, so I think a bit better default is to open in text mode on default), it doesn't work, it produces an access violation:
>
> // #1
> import std.stdio, std.array, std.algorithm;
> void main() {
>      auto lazyWords = File("words.txt", "r").byLine();
>      auto maxLen = reduce!max(map!"a.length"(lazyWords));
>      writeln(filter!((w){ return w.length == maxLen; })(lazyWords));
> }
>
>
> Finding the max length consumes the lazy iterable, so if the text file is small a possible functional style solution is to convert the lazy iterable into an array (other functional style solutions are possible, like reading the file twice, etc):
>
> // #2
> import std.stdio, std.array, std.algorithm;
> void main() {
>      auto lazyWords = File("words.txt", "r").byLine();
>      auto words = array(lazyWords);
>      auto maxLen = reduce!max(map!"a.length"(words));
>      writeln(filter!((w){ return w.length == maxLen; })(words));
> }
>
> But #2 doesn't work, because while "words" is an array of the words, byLine() uses the same buffer, so words is a sequence of useless stuff.
>
> To debug that code you need to dup each array item:
>
> // #3
> import std.stdio, std.array, std.algorithm;
> void main() {
>      auto lazyWords = File("words.txt", "r").byLine();
>      auto words = array(map!"a.dup"(lazyWords));
>      auto maxLen = reduce!max(map!"a.length"(words));
>      writeln(filter!((w){ return w.length == maxLen; })(words));
> }
>
>
> #3 works, but for scripting-line programs it's not the first version I write, I have had to debug the code. I think other programmers will have the same troubles. So I think byLine() default behavour is a bit bug-prone. Re-using the same buffer gives a good performance boost, but it's often a premature optimization in scripts. I prefer a language and library to use unsafe optimizations only on request.
>
> My suggested solution is to split File.byLine() in two member functions, one returns a lazy iterable that reuses the line buffer and one that doesn't reuse them. And my preferred solution is to give the shorter name (so it becomes the "default") to the method that dups the line, because this is the safer (less bug prone) behaviour. This is also in accord with D Zen. They may be named byLine() and byFastLine() or something similar.
>
> If you prefer the "default" one to be the less safe version, then the safe version may be named byDupLine():
>
> // #4
> import std.stdio, std.array, std.algorithm;
> void main() {
>      auto lazyWords = File("words.txt", "r").byDupLine();
>      auto lines = array(lazyWords);
>      auto maxLen = reduce!max(map!"a.length"(words));
>      writeln(filter!((w){ return w.length == maxLen; })(words));
> }
>
>
> Using the "maxs" (from http://d.puremagic.com/issues/show_bug.cgi?id=4705 ), the code becomes:
>
> // #5
> import std.stdio, std.algorithm;
> void main() {
>      auto lazyWords = File("words.txt", "r").byDupLine();
>      writeln(maxs!"a.length"(lazyWords));
> }

But you don't need a new string for each line to evaluate max over line 
lengths; the current byLine works.

Generally I think buffer reuse in byLine() is too valuable to let go.


Andrei