D and i/o

[Jonathan Marler]

On Sunday, 10 November 2019 at 19:41:52 UTC, Jon Degenhardt wrote:
> On Saturday, 9 November 2019 at 23:39:09 UTC, bioinfornatics 
> wrote:
>> [...]
>
> You might also be interested in a similar I/O performance test 
> I created: https://github.com/jondegenhardt/dcat-perf. This one 
> is based on 'cat' (copy to standard output) rather than 'wc', 
> as I'm interested in both input and output, but the general 
> motivation is similar. I specifically wanted to compare native 
> phobos facilities to those in iopipe and some phobos covers in 
> tsv-utils. Most tests are by-line based, as I'm interested in 
> record oriented operations, but chunk-based copying is included.
>
> A general observation is that if lines are involved, it's 
> important to measure performance of both short and long lines. 
> This may even affect 'wc' when reading by chunk or memory 
> mapped files, see H. S. Teoh's observations on 'wc' 
> performance: 
> https://forum.dlang.org/post/mailman.664.1571878411.8294.digitalmars-d@puremagic.com.
>
> As an aside - My preliminary conclusion is that phobos 
> facilities are overall quite good (based on tsv-utils 
> comparative performance benchmarks), but are non-optimal when 
> short lines are involved. This is the case for both input and 
> output. Both the tsv-utils covers and iopipe are better, with 
> iopipe being the best for input, but appears to need some 
> further work on the output side (or I don't know iopipe well 
> enough). By "preliminary", I mean just that. There could 
> certainly be mistakes or incomplete analysis in the tests.
>
> --Jon

For "cat" I believe there is a system call to tell the kernel to 
forward data from one file descriptor to the other, meaning you 
could implement cat without ever mapping the data into user-space 
at all. I'm sure this would be the fastest mechanism to implement 
cat, and I've seen this system call used by a version of cat 
somewhere out there.