csvReader & specifying separator problems...

[Jon Degenhardt]

On Thursday, 14 November 2019 at 12:25:30 UTC, Robert M. Münch 
wrote:
> Just trying a very simple thing and it's pretty hard: "Read a 
> CSV file (raw_data) that has a ; separator so that I can 
> iterate over the lines and access the fields."
>
> 	csv_data = raw_data.byLine.joiner("\n")
>
> From the docs, which I find extremly hard to understand:
>
> auto csvReader(Contents = string, Malformed ErrorLevel = 
> Malformed.throwException, Range, Separator = char)(Range input, 
> Separator delimiter = ',', Separator quote = '"')
>
> So, let's see if I can decyphre this, step-by-step by trying 
> out:
>
> 	csv_records = csv_data.csvReader();
>
> Would split the CSV data into iterable CSV records using ',' 
> char as separator using UFCS syntax. When running this I get:
>
> [...]

Side comment - This code looks like it was taken from the first 
example in the std.csv documentation. To me, the code in the 
std.csv example is doing something that might not be obvious at 
first glance and is potentially confusing.

In particular, 'byLine' is not reading individual CSV records. 
CSV can have embedded newlines, these are identified by CSV 
escape syntax. 'byLine' doesn't know the escape syntax. If there 
are embedded newlines, 'byLine' will read partial records, which 
may not be obvious at first glance. The .joiner("\n") step puts 
the newline back, stitching fields and records back together 
again in the process.

The effect is to create an input range of characters representing 
the entire file, using 'byLine' to do buffered reads. This input 
range is passed to CSVReader.

This could also be done using 'byChunk' and 'joiner' (with no 
separator). This would use a fixed size buffer, no searching for 
newlines while reading, so it should be faster.

An example:

==== csv_by_chunk.d ====
import std.algorithm;
import std.csv;
import std.conv;
import std.stdio;
import std.typecons;
import std.utf;

void main()
{
     // Small buffer used to show it works. Normally would use a 
larger buffer.
     ubyte[16] buffer;
     auto stdinBytes = stdin.byChunk(buffer).joiner;
     auto stdinDChars = stdinBytes.map!((ubyte b) => cast(char) 
b).byDchar;

     writefln("--------------");
     foreach (record; stdinDChars.csvReader!(Tuple!(string, 
string, string)))
     {
         writefln("Field 0: |%s|", record[0]);
         writefln("Field 1: |%s|", record[1]);
         writefln("Field 2: |%s|", record[2]);
         writefln("--------------");
     }
}

Pass it csv data without embedded newlines:

$ echo $'abc,def,ghi\njkl,mno,pqr' | ./csv_by_chunk
--------------
Field 0: |abc|
Field 1: |def|
Field 2: |ghi|
--------------
Field 0: |jkl|
Field 1: |mno|
Field 2: |pqr|
--------------

Pass it csv data with embedded newlines:

$ echo $'abc,"LINE 1\nLINE 2",ghi\njkl,mno,pqr' | ./csv_by_chunk
--------------
Field 0: |abc|
Field 1: |LINE 1
LINE 2|
Field 2: |ghi|
--------------
Field 0: |jkl|
Field 1: |mno|
Field 2: |pqr|
--------------

An example like this may avoid the confusion about newlines. 
Unfortunately, the need to do the odd looking conversion from 
ubyte to char/dchar is undesirable in a code example. I haven't 
found a cleaner way to write that. If there's a nicer way I'd 
appreciate hearing about it.

--Jon