std.stream replacement

[Timon Gehr]

On 03/05/2013 08:12 PM, Dmitry Olshansky wrote:
> ...
>
> There is one thing I found a nice abstraction while helping out on D's
> lexer in D and I call it mark-slice range. An extension to forward range
> it seems.
>
> It's all about buffering and defining a position in input such that you
> don't care for anything up to this point. This means that starting from
> thusly marked point stuff needs to be kept in buffer, everything prior
> to it could be discarded. The 2nd operation "slice" is getting a slice
> of some internal buffer from last mark to the current position.
> ...

The lexer I have built last year does something similar. It allows the 
parser to save and restore sorted positions in FIFO order with one 
size_t of memory inside the parser's current stack frame (internally, 
the lexer only saves the first position). The data is kept in a circular 
buffer that grows dynamically in case the required lookahead is too large.