Streaming transport interfaces: input

[Denis Koroskin]

On Fri, 15 Oct 2010 05:49:05 +0400, Andrei Alexandrescu  
<SeeWebsiteForEmail at erdani.org> wrote:

> On 10/14/2010 07:57 PM, Denis Koroskin wrote:
>> In my original version there was a setbuf(f, null) call. I removed it
>> because it had 0 impact on performance.
>> I also tried using unistd open/read functions, that had zero impact, too
>
> Yah, I looked more into the behavior of fread. At least on a couple of  
> implementations that offer source, the implementation first copies  
> whatever data is (if any) in the internal buffer, and then read the rest  
> of the data straight into the remaining user-supplied buffer.
>
> This means, if you consistently fread into a buffer, the internal  
> buffers are never touched.
>
> I stand corrected regarding the extra copy.
>
>> (btw, opening file with O_DIRECT returned valid file descriptor, but
>> read operations very failing with an invalid argument error).
>
> I recall I've read a discussion where Linus was really down on O_DIRECT.  
> Found it: http://kerneltrap.org/node/7563
>
>
> Andrei

Yeah, I've read that.

Btw, I've re-run my tests, O_DIRECT works but also has no effect. Best of  
3 results for parsing 33Mb log file:

using getline (C++ -O2):
real 0m0.081s
user 0m0.068s
sys 0m0.016s

using byLine (D -O -release -inline):
real 0m0.067s
user 0m0.056s
sys 0m0.012s

That's a 20% difference. Source and test files here:
http://rapidshare.com/files/425154119/tests.7z  309k
http://rapidshare.com/files/425154408/tests.zip 6Mb (in case you have no  
7z installed)