iopipe v0.2.0 - safe update

[aberba]

On Monday, 29 June 2020 at 11:58:10 UTC, Steven Schveighoffer 
wrote:
> On 6/28/20 8:37 PM, aberba wrote:
>> On Sunday, 28 June 2020 at 20:26:43 UTC, JN wrote:
>>> What's iopipe and what does it do? How does it compare with 
>>> std.process?
>> 
>> I my line of words, its what you'd use to stream large files 
>> and do processing on it. Like CSV, video??, Json, and the 
>> like. Its high performance cus it optimized to haven great 
>> performance even for large files. Designed to let you chain 
>> the processing pipeline using reusable functions...and  you 
>> can compose one yourself.
>
> Yes, the emphasis is on being able to handle large sets of data 
> similarly to how you would handle an array, but not having to 
> load the whole thing into an array. It uses buffering to 
> present an array-like interface. The only difference between it 
> and arrays is you have to tell it how much data is important to 
> you at the moment.
>
> There is also a focus on composability -- I want you to be able 
> to build any type of harness to take what you have and turn it 
> into something you can feed into a parser/handler.
Thanks for the expansion. I could really use this for an article. 
Added to my to-do list.


>
> Take for example, jsoniopipe [1], which is a library that uses 
> iopipe to parse JSON. You can feed it any source -- a string, a 
> zipped file, a network socket, and all you have to do is 
> compose the pieces to get it to be a text iopipe (an iopipe 
> where the window type is a string type). Then the json library 
> will parse it and provide you with a stream of JSON data. 
> Because everything is available to the compiler, it can 
> optimize/inline across all the layers, and provide the best 
> performance possible (see my presentation at dconf 2017 [2] for 
> how this works out, also linked by Clarice).
Will looking into that library more.

> and all you have to do is compose the pieces to get it to be a 
> text iopipe (an iopipe where the window type is a string type).
No sure I understand this statement.

>
>> 
>> I'm curious myself how it differs from the NodeJs Stream API. 
>> Would you say iopipe is a Streaming API by definition?
>
> I'm not familiar with NodeJs in general so I can't comment on 
> that. I would say iopipe is focused on making streaming more 
> like dealing with arrays, with as high performance as possible. 
> One thing I've always disliked about most streaming libraries 
> is that they hide the buffer as an implementation detail, but 
> it should be the focus.
How well does iopipe work in a multi-threaded environment?

>
> -Steve
>
> [1] https://code.dlang.org/packages/jsoniopipe
> [2] https://www.youtube.com/watch?v=un-bZdyumog
I'll probably re-watch it before writing the article.