Parallel ByLine, ByChunk?

[dsimcha]

On 8/9/2011 3:14 PM, Graham Fawcett wrote:
> Interesting. A possible use-case would be reading from a set of files,
> say, a directory full of log files which need to be analyzed
> line-by-line. So a reader that accepted a range of files might be
> handy. (I guess that the file concatenation could be handled at the OS
> level, with "cat" and a pipe, but a range-of-files would be more
> convenient.)
>
> On a tangent, the first thing that came to mind when you mentioned
> this was Tim Bray's WideFinder benchmark:
>
>    http://wikis.sun.com/display/WideFinder/Wide+Finder+Home
>
> ParallelByLine would make for a great WideFinder implementation.
>
> Best,
> Graham
>

I made a pull request 
(https://github.com/D-Programming-Language/phobos/pull/179) for another 
overload of asyncBuf in std.parallelism that will make parallel byLine 
or byChunk trivial to implement (reading lines in parallel is actually 
the documentation example).  I'm not sure whether it's worth adding an 
actual ParallelByLine to std.stdio, given how trivial the new asyncBuf 
overload makes it.  At any rate, I'd rather get the lower level 
infrastructure finalized and into Phobos before I build the higher level 
stuff on top of it.