Parsing and splitting textfile

[Justin Whear]

On Mon, 24 Feb 2014 15:19:06 -0500, Hugo Florentino wrote:

> On Mon, 24 Feb 2014 19:08:16 +0000 (UTC), Justin Whear wrote:
>>
>> Specifically std.regex.splitter[1] creates a lazy range over the input.
>> You can couple this with lazy file reading (e.g.
>> `File("mailbox").byChunk (1024).joiner`).
>>
>>
> Would something like this work? (I cannot test it right now)
> 
> auto themailbox = args[1];
> immutable uint chunksize = 1024 * 64;
> static auto re = regex(`\n\nFrom .+ at .+$`);
> auto mailbox;
> auto mail;
> while (mailbox = File(themailbox).byChunk(chunksize).joiner) != EOF) {
>    mail = splitter(mailbox, re);
> }
> 
> If so, I have a couple of furter doubts:
> 
> Using splitter actually removes the expression from the string, how
> could I reinsert it to the beginning of each resulting string in an
> efficient way (i.e. avoiding copying something which is already loaded
> in memory)?
> 
> I am seeing the splitter fuction returns a struct, how could I
> progressively dump to disk each resulting string, removing it from the
> struct, so that so that it does not end up having the full mailbox
> loaded into memory, in this case as a struct?
> 
> Regards, Hugo

The code you've posted won't work, primarily because you don't need to 
loop over the file-reading range, nor will it ever "return" EOF.  Also, 
if you don't actually want to remove the regex matches, you can just use 
the matchAll function.  Here's some _untested_ sample code to set you on 
the right track.

--------------------------------------------------------
import std.algorithm, std.range, std.stdio, std.regex;
void main(string[] args)
{
    auto mailboxPath = args[1];
    immutable size_t chunksize = 1024 * 64;
    auto re = regex(`\n\nFrom .+ at .+$`);
    // you might want to try using ctRegex

    auto mailStarts = File(mailboxPath).byChunk(chunksize).joiner
                          .matchAll(re);
}
--------------------------------------------------------

This code won't actually do any work--no data will be loaded from the 
file (caveat: the first chunk might be prefetched, not sure), no matches 
will actually be performed.  If you use `take(10)` on the mailStarts 
variable, the code will load only as much of the file (down to the 
granularity of chunksize) as is needed to find the first 10 instances of 
the regular expression.  The regex matches will not copy, but rather 
provide slices over the data that is in memory.

And, thinking about this further, you don't want to use my code either--
partly because byChunk reuses its buffer, partly because the functions in 
std.regex provide slices over the input data.  I think what you'll want 
to do is: load the data from File chunk-by-chunk lazily, scan each chunk 
with the regex, if you don't find a match, copy that data into an 
"overlap" buffer and repeat, if you do find a match then the contents of 
the overlap buffer + the slice up to the current match is one mail, rinse 
and repeat.  You should be able to encapsulate all of this in clean, lazy 
range, but I don't have the time right now to work out if it can be done 
by simply compositing existing functions from Phobos.

Justin