D and i/o

[Jonathan Marler]

On Sunday, 10 November 2019 at 07:16:31 UTC, bioinfornatics wrote:
> On Saturday, 9 November 2019 at 23:39:09 UTC, bioinfornatics 
> wrote:
>> Dear,
>>
>> In my field we are io bound thus I would like to have our 
>> tools fast as I can read a file.
>>
>> Thus I started some dummy bench which count the number of 
>> lines.
>> The result is compared to wc -l command. The line counting is 
>> only a pretext to evaluate the io, this process can be 
>> switched by any io processing. Thus we use much as possible 
>> the buffer instead the byLine range. Moreover such range imply 
>> that the buffer was read once before to be ready to process.
>>
>>
>> https://github.com/bioinfornatics/test_io
>>
>> Ideally I would like to process a shared buffer through 
>> multiple core and run a simd computation. But it is not yet 
>> done.
>
> If you have some scripts or enhancements you are welcome
>
> Currently results show that naïve implementation is at least 
> twice time slower than wc, up to 5 slower for // scripts

Here's an example implementation of wc using mmap:

#!/usr/bin/env rdmd
import std.stdio, std.algorithm, std.mmfile;

void main(string[] args)
{
     foreach (arg; args[1..$])
     {
         auto file = new MmFile(arg, MmFile.Mode.read, 0, null);
         auto content = cast(char[])file.opSlice;
         writefln("%s", content.count('\n'));
     }
}