iopipe v0.0.4 - RingBuffers!

[Dmitry Olshansky]

On Thursday, 10 May 2018 at 23:22:02 UTC, Steven Schveighoffer 
wrote:
> OK, so at dconf I spoke with a few very smart guys about how I 
> can use mmap to make a zero-copy buffer. And I implemented this 
> on the plane ride home.
>
> However, I am struggling to find a use case for this that 
> showcases why you would want to use it. While it does work, and 
> works beautifully, it doesn't show any measurable difference 
> vs. the array allocated buffer that copies data when it needs 
> to extend.

I’d start with something clinicaly synthetic.
Say your record size is exactly half of buffer + 1 byte. If you 
were to extend the size of buffer, it would amortize.

Basically:
16 Mb buffer fixed
vs
16 Mb mmap-ed ring

Where you read pieces in 8M+1 blocks.Yes, we are aiming to blow 
the CPU cache there. Otherwise CPU cache is so fast that 
ocasional copy is zilch, once we hit primary memory it’s not. 
Adjust sizes for your CPU.

The amount of work done per byte though has to be minimal to 
actually see anything.


> 
> in the buffer. But alas, it's roughly the same, even with large 
> number of lines for context (like 200).
>
> However, this example *does* show the power of iopipe -- it 
> handles all flavors of unicode with one template function, is 
> quite straightforward (though I want to abstract the line 
> tracking code, that stuff is really tricky to get right). Oh, 
> and it's roughly 10x faster than grep, and a bunch faster than 
> fgrep, at least on my machine ;) I'm tempted to add regex 
> processing to see if it still beats grep.

Should be mostly trivial in fact. I mean our first designs for 
IOpipe is where I wanted regex to work with it.

Basically - if we started a match, extend window until we get it 
or lose it. Then release up to the next point of potential start.

>
> Next up (when my bug fix for dmd is merged, see 
> https://issues.dlang.org/show_bug.cgi?id=17968) I will be 
> migrating iopipe to depend on 
> https://github.com/MartinNowak/io, which should unlock Windows 
> support (and I will add RingBuffer Windows support at that 
> point).
>
> Enjoy!
>
> https://github.com/schveiguy/iopipe
> https://code.dlang.org/packages/iopipe
> http://schveiguy.github.io/iopipe/
>
> -Steve