Parallel ByLine, ByChunk?

[Graham Fawcett]

On Tue, 09 Aug 2011 18:58:22 +0000, dsimcha wrote:
>
> Is there any interest in a ParallelByChunk and ParallelByLine in
> std.stdio.File? These would have a set of some user specified size
> nBuffers of buffers, which would be recycled. A background thread
> would read ahead until all buffers are full, then wait on a
> condition variable. When popFront() has been called several times
> and a large number of buffers are available, the background thread
> would be woken to fill the newly available buffers.

Interesting. A possible use-case would be reading from a set of files,
say, a directory full of log files which need to be analyzed
line-by-line. So a reader that accepted a range of files might be
handy. (I guess that the file concatenation could be handled at the OS
level, with "cat" and a pipe, but a range-of-files would be more
convenient.)

On a tangent, the first thing that came to mind when you mentioned
this was Tim Bray's WideFinder benchmark:

  http://wikis.sun.com/display/WideFinder/Wide+Finder+Home

ParallelByLine would make for a great WideFinder implementation.

Best,
Graham