etc.curl: Formal review begin

Marco Leise Marco.Leise at gmx.de
Tue Aug 30 14:11:49 PDT 2011


Am 30.08.2011, 21:56 Uhr, schrieb Walter Bright  
<newshound2 at digitalmars.com>:

> On 8/30/2011 11:59 AM, Andrei Alexandrescu wrote:
>> Unless the OS issues speculative reads (which I don't think it does for  
>> either
>> files or sockets), any time spent in write() is a net loss for reading  
>> speed.

Now *that* was speculative, if you know what I mean. Take a look at this  
article on on-demand readahead: http://lwn.net/Articles/235164/
I suspect the situation on other operating-systems is similar.
The buffer for Linux pipes is currently maxed at 64 KB, so there is some  
room for multiprocessing as well.
Sockets are different, but as someone else already pointed out, the OS has  
to handle the incoming traffic somehow, and it does so by filling a  
receive buffer. The socket implementation in current operating systems  
allows you to set the buffer size to large values. The network protocol  
practically enforces asynchronous operations in the background.

>> Now, if write is buffered and the buffers are flushed asynchronously,  
>> calls to
>> write() would be instantaneous. I'm not sure to what extent the major  
>> OSs do
>> that, and for what types of files.

Both OSs and disk drives have write caches. For removable media the OS  
cache is often disabled to reduce data loss. The on-disk cache is usually  
several MB in size so small files can be transferred at full bus speed. On  
SATA systems the disk can then decide to reorder read and write commands  
for optimal performance (http://www.sata-io.org/technology/ncq.asp). The  
filesystem driver in the kernel may want to stop the drive from messing  
around with the command order, because it wants to perform some atomic  
operation so it is also still possible to wait for the drive flush its  
write buffer. This is a trade-off between performance and security.

> The way to test this is to read file by 4096 byte chunks forwards, and  
> compare that with the speed of reading it backwards.
>
> The OS will try to cache the file in memory between runs, so to defeat  
> this means the file has to be on a removable drive (I use an SD card)  
> and pull the drive out and reinsert it between each test.

On Linux you can clear the caches with "echo 3 > /proc/sys/vm/drop_caches"  
(kernel 2.6.16)
For Windows I could only find this side-effect of opening a file in  
unbuffered mode on WinXP:  
http://stackoverflow.com/questions/478340/clear-file-cache-to-repeat-performance-testing/7113153#7113153

- Marco


More information about the Digitalmars-d mailing list