etc.curl: Formal review begin

[Johannes Pfau]

Andrei Alexandrescu wrote:
>On 8/30/11 1:21 PM, jdrewsen wrote:
>> Den 30-08-2011 19:38, Andrei Alexandrescu skrev:
>>> On 8/30/11 12:22 PM, jdrewsen wrote:
>>>> Walter suggested that I should write an article about using the
>>>> wrapper. I've now taken the first steps on writing such an
>>>> article. I will have to get the library API rock stable before I
>>>> can finish it though.
>>>
>>> I have a suggestion for you - write and test an asynchronous copy
>>> program.
>>>
>>> It is a continuous source of surprise to me that even seasoned
>>> programmers don't realize that this is an inefficient copy routine:
>>>
>>> while (read(source, buffer))
>>> write(target, buffer);
>>>
>>> If the methods are synchronous and the speeds of source and target
>>> are independent, the net transfer rate of the routine is
>>> R1*R1/(R1+R2), where R1 and R2 are the transfer rates of the source
>>> and destination respectively. In the worst case R1=R2 and the net
>>> transfer rate is half that.
>>>
>> [snip]
>>
>> I guess that e.g. incoming network buffers in the OS often makes the
>> shown copy routine faster than you would think i most cases. These
>> buffers stores the incoming network data asynchronously by the OS and
>> makes the next read() instantanous. The same can be said about
>> writing to disk. Calling sync() is the real enemy here. This is only
>> true as long as the buffers are not full of course.
>
>Unless the OS issues speculative reads (which I don't think it does
>for either files or sockets), any time spent in write() is a net loss
>for reading speed. Now, if write is buffered and the buffers are
>flushed asynchronously, calls to write() would be instantaneous. I'm
>not sure to what extent the major OSs do that, and for what types of
>files.
>
>
>Andrei

Write buffering / asynchronous buffer flushing happens at least in
Linux and Windows. This is the main reason for the windows "safe-eject"
function for usb-sticks. And linux/gnome does a similar thing when
ejecting a usb drive; it even shows a "writing data to disk"
progressbar. It's also the reason why there's a O_SYNC flag for the
posix "open" method.

Here's a snipped that proves that sockets also do asynchronous flushing:
https://gist.github.com/1182041
Note that you can even exit the client before the server ever makes
it's first receive call, and it'll still get the data.

To really test if the data is buffered at client or server-side, you'd
need two machines and cut the connection at the right moment.
-- 
Johannes Pfau