etc.curl: Formal review begin

dsimcha dsimcha at yahoo.com
Wed Aug 31 14:05:16 PDT 2011


== Quote from Johannes Pfau (spam at example.com)'s article
> Andrei Alexandrescu wrote:
> >
> >This will happen all the more if you have multiple threads.
> >
> >You clearly have a good expertise on OS, I/O, and related matters, and
> >I am honoured by your presence and by the value you're adding to this
> >group. However, in this particular argument you're only firing blanks.
> >Are you sure you have a case?
> >
> >
> >Andrei
> So why don't we benchmark this?
> Here's a first, naive attempt:
> https://gist.github.com/1184671
> There's not much to do for the sync implementation, but the async one
> should maybe keep a pool of buffers and reuse those. However, the async
> implementation was already ~10% faster in simple tests (copying a 700mb
> ubuntu .iso; source and target on the same harddisk). I wouldn't have
> expected this, but it seems the async copy is actually be faster.

Wow, that is naive.  You're creating a new buffer for every chunk.  I'm guessing
that GC overhead eats you alive.  The next Phobos release will have a new overload
of std.parallelism.asyncBuf that will make it easy to recycle buffers in these
cases.  If DMD Bug 6582 gets fixed or I find a workaround, I'll also make a
better-encapsulated std.stdio.File.ByChunkAsync on top of this.  This would make
asynchronous file copying without an absurd number of allocations trivial.  It
would be something like:

void copy(in char[] source, in char[] dest) {
    // Pray that all the d'tor bugs in DMD are fixed so these files
    // get closed automatically.

    auto destHandle = File(dest, "wb");
    foreach(chunk; File(source).byChunkAsync(chunkSize)) {
        destHandle.rawWrite(chunk);
    }
}


More information about the Digitalmars-d mailing list