iopipe v0.1.0 - now with Windows support!

[Steven Schveighoffer]

On 6/12/18 1:51 AM, DigitalDesigns wrote:
> 
> Could you explain some benefits specific to this implementation and a 
> bit of the functional aspects for a proper overview of it's capabilities 
> and why I should chose this method over others?

The things that I think make this approach better are:

1. Direct buffer access

Direct buffer access I have found is one of those ideas that doesn't 
seem like it's that impressive until you start using it. Many times, 
buffering is used in a generic library for optimization (basically 
amortizing reads) and is an implementation detail hidden from your view. 
Think of how FILE * keeps a large buffer of data inside itself, but only 
gives you access to one character at a time.

This forces you to create your *own* buffering scheme on top of that. 
What a waste! Iopipe allows you to use buffering for your purposes on 
top of the benefits of amortization. It's my belief that this is why 
iopipe's byline feature is 2x faster than Phobos'.

2. Using templates to their fullest

Iopipes are all templated on the buffer or iopipe underneath it. This 
makes tings easily swappable. It's really cool to be able to take your 
JSON or XML parser, and hook it onto an in-memory string in one line, 
and then hook it onto a socket, and everything is optimized for that 
situation. It takes the fun and flexibility of range programming and 
brings it to i/o.

This is why iopipe's byline handles all forms of UTF, compared to Phobos 
which only handles UTF8.

For example, I handle all forms of UTF with iopipe, with a decent set of 
utilities. Here is a complete program using iopipe that converts any 
type of UTF into another type, optimized for the specific situation:

https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d

3. Compiler optimization for everything

All parts of iopipe, except for the low-level reads and writes (which 
ironically are not really part of iopipe) are visible to the compiler 
for inlining and optimization. I'm leveraging the power of the decades 
of optimization experience that the compiler can provide. This makes it 
easy to write code that performs well.

An anecdote: For my talk on iopipe in 2017 
(http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live 
demo showing the performance power. I literally was still working on it 
2 or 3 days before, while at dconf. I had already written a JSON parser, 
which was part of my presentation, but when I was showing it to another 
D user (Daniel Murphy), I couldn't really answer the question "how does 
it perform?". So he gave me a challenge -- do pretty printing on a JSON 
file. Simple enough, with the parser I had already written, took me 
about 1 hour to write it. It performed poorly compared to what we would 
have expected, but tweaking a few things (almost all were due to using 
some algorithms incorrectly), I got it to go faster than RapidJson in 
certain use cases, and reasonably close in others. And I did nothing in 
terms of lookup tables or using any special instructions. All in all, it 
was probably 2 hours of work, and the code is beautiful IMO! 
https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d

I think anyone who is doing parsing should have a look at iopipe, it not 
only may make your code much simpler and easier to read and write, but 
it would help me tune iopipe to cater to parsers, which I think is its 
wheelhouse.

I plan to eventually finish the JSON parser for a releasable state, and 
eventually tackle XML and a few other things.

-Steve