D and i/o

[Jon Degenhardt]

On Sunday, 10 November 2019 at 20:33:35 UTC, Jonathan Marler 
wrote:
> On Sunday, 10 November 2019 at 19:41:52 UTC, Jon Degenhardt 
> wrote:
>> On Saturday, 9 November 2019 at 23:39:09 UTC, bioinfornatics 
>> wrote:
>>> [...]
>>
>> You might also be interested in a similar I/O performance test 
>> I created: https://github.com/jondegenhardt/dcat-perf. This 
>> one is based on 'cat' (copy to standard output) rather than 
>> 'wc', as I'm interested in both input and output, but the 
>> general motivation is similar. I specifically wanted to 
>> compare native phobos facilities to those in iopipe and some 
>> phobos covers in tsv-utils. Most tests are by-line based, as 
>> I'm interested in record oriented operations, but chunk-based 
>> copying is included.
>>
>> [...]
>
> For "cat" I believe there is a system call to tell the kernel 
> to forward data from one file descriptor to the other, meaning 
> you could implement cat without ever mapping the data into 
> user-space at all. I'm sure this would be the fastest mechanism 
> to implement cat, and I've seen this system call used by a 
> version of cat somewhere out there.

Thanks, I wasn't aware of this. But perhaps I should describe the 
motivation in more detail. I'm not actually interested in 'cat' 
per se, it is just a stand-in for the more general processing I'm 
typically interested in. In every case I'm operating on the 
records in some form (lines or something else), making a 
transformation, and depending on application, writing something 
out. This is the case in tsv-utils as well as many scenarios of 
the systems I work on (search engines). These applications 
sometimes operate on data streams, sometimes on complete files. 
Hence my interest in line-oriented I/O performance.

Obviously there is a lot more ground in the general set of 
applications I'm interested in than is covered in the simple 
performance tests in dcat-perf, but it's a starting point. It's 
also why I didn't make comparisons to existing versions of 'cat'.