stream == range ? [Sliding window]

[Dragos Carp via Digitalmars-d]

>>> Hardly. In fact, I've spent quite some time trying to figure 
>>> this out.
>>>
>>> Doing bulk read to some array is the pushing burden of 
>>> maintaining
>>> some buffer on the user and adding the overhead of extra copy 
>>> on
>>> buffered streams. Not to mention that the more methods we put 
>>> in range
>>> API the more if/else forests we produce in our algorithms.
>>
>> LinearBuffer tries to address this problem in:
>> https://github.com/D-Programming-Language/phobos/pull/2928
>> This simply extends the existing std.array.Appender with 
>> popFrontN
>> operations.
>>
>> The use case would be like this:
>> 1. accumulate data in the LiniearBuffer by appending to the 
>> buffer
>> 2. try to identify the next token
>> 3. if a token is identified popFontN the length of the token
>> 4. otherwise acumulate some more (go to 1.)
>>
>
> Yes, that's the pattern. Thoughts:
> 1. popFrontN is just confusing - this is not a range, let's not 
> pretend it is.

LinearBuffer is not meant as replacement of streams or such. It 
is a low level data structure, that maybe could be used in the 
implementation of the streams.
Considering the FIFO style of usage, I found popFrontN 
appropriate. But it would be no problem to rename it.

> 2. As it stands it would require first copying into appender 
> which is slow. The opposite direction is usually a better match 
> - assume appender (or rather buffer) is filled to its capacity 
> with data from the file and keeping this invariant true.

This would function differently if the data comes over the 
socket, usually you cannot fill the buffer.

>
> Otherwise it looks quite similar to the sliding window 
> interface and that is great.
>
> data --> window
> popFrontN --> skip
> put ~~> extend (probably should be just called 'more')

Sometimes (socket recv) you don't know how much extend.

>
> Actually I've dig up my original code for buffer, amended and 
> integrated in one of Steven's Schveighoffer I/O package forks:
>
> https://github.com/DmitryOlshansky/phobos/blob/new-io3/std/io/textbuf.d#L68
> Now I recall it's not quite sliding window as it shrinks on 
> reads...
> Anyhow the same pattern goes with: data, discard and extend.
>
>> I was also thinking about adding a new method:
>> ptrdiff_t putFrom(ptrdiff_t delegate(T[]) read);
>> meant to be used together with file read or socket receive 
>> functions. In
>> this way the read/recv system functions can write directly 
>> into the
>> input buffer without intermediate copies.
>
> Yes, that is the missing piece in the current setup, something 
> got to fill the buffer directly.
>
> Parser or any other component down the line shouldn't concern 
> itself with reading, just using some "more" primitive to 
> auto-magically extend the window by at least X bytes. That 
> would shrink-extend buffer  and load it behind the scenes.

As I sad, LinearBuffer is meant to be used "behind the scenes".

I find Andrei's bulkRead, maybe combined with a matcher a la 
boost::asio read_until, the simplest high level interface.