iopipe v0.2.0 - safe update

[Steven Schveighoffer]

On 6/28/20 8:37 PM, aberba wrote:
> On Sunday, 28 June 2020 at 20:26:43 UTC, JN wrote:
>> What's iopipe and what does it do? How does it compare with std.process?
> 
> I my line of words, its what you'd use to stream large files and do 
> processing on it. Like CSV, video??, Json, and the like. Its high 
> performance cus it optimized to haven great performance even for large 
> files. Designed to let you chain the processing pipeline using reusable 
> functions...and  you can compose one yourself.

Yes, the emphasis is on being able to handle large sets of data 
similarly to how you would handle an array, but not having to load the 
whole thing into an array. It uses buffering to present an array-like 
interface. The only difference between it and arrays is you have to tell 
it how much data is important to you at the moment.

There is also a focus on composability -- I want you to be able to build 
any type of harness to take what you have and turn it into something you 
can feed into a parser/handler.

Take for example, jsoniopipe [1], which is a library that uses iopipe to 
parse JSON. You can feed it any source -- a string, a zipped file, a 
network socket, and all you have to do is compose the pieces to get it 
to be a text iopipe (an iopipe where the window type is a string type). 
Then the json library will parse it and provide you with a stream of 
JSON data. Because everything is available to the compiler, it can 
optimize/inline across all the layers, and provide the best performance 
possible (see my presentation at dconf 2017 [2] for how this works out, 
also linked by Clarice).

> 
> I'm curious myself how it differs from the NodeJs Stream API. Would you 
> say iopipe is a Streaming API by definition?

I'm not familiar with NodeJs in general so I can't comment on that. I 
would say iopipe is focused on making streaming more like dealing with 
arrays, with as high performance as possible. One thing I've always 
disliked about most streaming libraries is that they hide the buffer as 
an implementation detail, but it should be the focus.

-Steve

[1] https://code.dlang.org/packages/jsoniopipe
[2] https://www.youtube.com/watch?v=un-bZdyumog