etc.curl: Formal review begin

[Sean Kelly]

On Aug 30, 2011, at 10:38 AM, Andrei Alexandrescu wrote:
> 
> This is an equation very easy to derive from first principles but many people are very incredulous about it. Consequently, many classic file copying programs (including cp; I don't know about wget or curl) use the inefficient method. As the variety of data sources increases (SSD, magnetic, networked etc) I predict async I/O will become increasingly prevalent. In an async approach with a queue, transfer proceeds at the optimal speed min(R1, R2). That's why I'm insisting the async range should be super easy to use, encapsulated, and robust: if people reach for the async range by default for their dealings with networked data, they'll write optimal code, sometimes even without knowing it.
> 
> If your article discusses this and shows e.g. how to copy data optimally from one server to another using HTTP, or from one server to a file etc, and if furthermore you show how your API makes all that a trivial five-liner, that would be a very instructive piece.

HTTP really isn't the ideal protocol for this sort of thing.  By default, the entire file would be in the message body, which isn't really asynchronous.  Chunking is an alternative, but there's no way to do this from the client-side, it's purely a server thing.  In short, HTTP kind of stinks for the client uploading large amounts of data to the server.  On a more general note though, I absolutely agree about AIO.