etc.curl: Formal review begin

[Marco Leise]

Am 31.08.2011, 00:11 Uhr, schrieb Andrei Alexandrescu  
<SeeWebsiteForEmail at erdani.org>:

> On 8/30/11 4:11 PM, Marco Leise wrote:
>> Am 30.08.2011, 21:56 Uhr, schrieb Walter Bright
>> <newshound2 at digitalmars.com>:
>>
>>> On 8/30/2011 11:59 AM, Andrei Alexandrescu wrote:
>>>> Now, if write is buffered and the buffers are flushed asynchronously,
>>>> calls to
>>>> write() would be instantaneous. I'm not sure to what extent the major
>>>> OSs do
>>>> that, and for what types of files.
>>
>> Both OSs and disk drives have write caches.
>
> That's not the question. The question is whether the caches are flushed  
> asynchronously and under what regime. What I can say as an end user of  
> fwrite() et al is that such calls do take time - it's not  
> near-instantaneous.
>
> Andrei

Ok, at least you added an anonymous 'regime' that may influence the buffer  
flushes ;)
To be honest, I think we can short-circuit the discussion. Your  
multi-threaded solution will perfectly hide away the delays between  
consecutive reads() and writes() in a file copy operation. Period. In  
other words, the slowest operation will be the mark.

What I want to show you is that your assumption, that an operating system  
doesn't flush asynchronously is wrong under the regime that you use the  
default settings when writing data to a local disk with the system specs I  
mention below. It does not mean that write() has to be instantaneous. We  
don't know what write() actually has to care for internally. I could  
imagine amongst other things: query and lock enough free disk space from  
the fs driver, check auxiliary modules (disk quota, encryption,  
compression), write a duplicate of the data to the disk/file cache and  
remove other data if it is full.

As a test case I wrote a small benchmark to get some numbers. It compares  
the default write() operation with two ways to get the synchronous  
behavior you predicted. The test runs the following steps:
- create an empty "test.file"
- flush pending I/O buffers to disk using the sync() system call, in order  
to ensure equal conditions.
- open the file write-only
- start timing
- write 1 MB of repeated 0..255 patterns
- close the file
- stop timing

The two synchronous test cases work like this:
A) open the file to be written in O_SYNC mode, making the write operation  
itself synchronous
B) use fsync() after the write operation

My test system is:
CPU: Core 2 Duo 2.0 Ghz
disk: WDC WD2500BEVS-75UST0 (SATA, UDMA-6, 5400 RPM, 8 MB cache)
filesystem: ext4
kernel: 2.6.39

The results show that caches make the write ~30 times faster:
default:   3_184 μs
O_SYNC : 104_572 μs
fsync(): 104_296 μs

To check back I remounted my root partition with the 'sync' option ("mount  
-o remount,rw,sync /"), effectively bypassing caches and ran the test  
again:
default: 109_361 μs
O_SYNC : 111_370 μs
fsync(): 102_295 μs

Actually "mount -o remount,rw,sync /" works without reboot and gives a  
nice impression of how the OS would feel without caches. Please try it if  
you have a Linux.
The benchmark code is in the attachment.

- Marco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cache_bench.d
Type: application/octet-stream
Size: 1381 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20110831/ad23b845/attachment-0001.obj>