Stream-Based Processing of Range Chunks in D

[Philippe Sigaud]

On Tue, Dec 10, 2013 at 10:57 AM, "Nordlöw" <per.nordlow at gmail.com> wrote:
> I'm looking for an elegant way to perform chunk-stream-based processing of
> arrays/ranges. I'm building a file indexing/search engine in D that
> calculates various kinds of statistics on files such as histograms and
> SHA1-digests. I want these calculations to be performed in a single pass
> with regards to data-access locality.

> Seemingly this is not a very elegant (functional) approach as I have to
> spread logic for each statistics (reducer) across three different places in
> the code, namely `start`, `put` and `finish`.

Concerning the put, you could have an auxiliary function that's
defined only once:

void delegate( /*typeofChunk?*/ chunk) worker, sha, bhist8;

if (doSHA1)
    sha = (chunk) { sha1.put(chunk);}
else
    sha = (chunk) {}

if (doBhist8)
    bhist8 = (chunk) { /*some BHist8 work*/}
else
    bhist8 = (chunk) {}

worker = (chunk) { sha(chunk); bist8(chunk};}

...

       foreach (chunk; src.chunks(chunkSize))
            worker(chunk);