etc.curl: Formal review begin

[Sean Kelly]

On Aug 30, 2011, at 11:59 AM, Andrei Alexandrescu wrote:

> On 8/30/11 1:21 PM, jdrewsen wrote:
>> 
>> 
>> I guess that e.g. incoming network buffers in the OS often makes the
>> shown copy routine faster than you would think i most cases. These
>> buffers stores the incoming network data asynchronously by the OS and
>> makes the next read() instantanous. The same can be said about writing
>> to disk. Calling sync() is the real enemy here. This is only true as
>> long as the buffers are not full of course.
> 
> Unless the OS issues speculative reads (which I don't think it does for either files or sockets), any time spent in write() is a net loss for reading speed. Now, if write is buffered and the buffers are flushed asynchronously, calls to write() would be instantaneous. I'm not sure to what extent the major OSs do that, and for what types of files.

They do this for writing to sockets, so I'd guess it applies to most types of files.  However, the write buffer is finite in size, and writing more than this amount will cause write to block.  The really great part is that I'm not aware of a way to determine how much space is available in the write buffer… only that a nonzero amount is available.  Moving data efficiently in either direction between locations with different transfer rates is a pretty tricky problem, and I don't know that the facilities exist to do this as well as one would like.