Streaming parsers in D a novel design

[ikod]

On Tuesday, 18 April 2023 at 08:13:05 UTC, Dmitry Olshansky wrote:
> A streaming parser is doing 2 things:
> - waiting for input
> - processing the current input and spitting out Events
>
> This maps to D beautifully:
>
> 1. Waiting for input means it's an OutputRange with explicit 
> put method!
>
> 2. Processing events means it's an InputRange of events. It may 
> even be ForwardRange, mine first of this kind is Forward range.

This is how `requests` handle input stream from the remote end 
and convert it to "input stream" of http response body chunks.

Input is a stream of bytes, arbitrary splitted by remote end or 
by network, etc. Moreover,response content can be gzipped, 
chunk-encoded and whatever else. So you have to process each 
network input with several "processors", like "inflate", 
"chunk-encoded decoder" and so on.

       +------------+   +--------------+   +---------+   
+-----------------+
       |input range |   | chunk-encoded|   |   gzip  |   | input 
range     |
Net->-| of byte[]  |->-|   decoder    |->-| decoder |->-| of 
response body|->- App
       |            |   |              |   |         |   | chunks: 
byte[]  |
       +------------+   +--------------+   +---------+   
+-----------------+

There is some problems like push all unprocessed data to the 
right end of the picture at the end of the network stream and 
maybe some other... But essentially it works.

Immutability of underlying data on every step of this processing 
stream helps very much.