Idiomatic way to write a range that tracks how much it consumes

[Steven Schveighoffer]

On 4/26/20 11:38 PM, Jon Degenhardt wrote:
> I have a string that contains a sequence of elements, then a terminator 
> character, followed by a different sequence of elements (of a different 
> type).
> 
> I want to create an input range that traverses the initial sequence. 
> This is easy enough. But after the initial sequence has been traversed, 
> the caller will need to know where the next sequence starts. That is, 
> the caller needs to know the index in the input string where the initial 
> sequence ends and the next sequence begins.
> 
> The values returned by the range are a transformation of the input, so 
> the values by themselves are insufficient for the caller to determined 
> how much of the string has been consumed. And, the caller cannot simply 
> search for the terminator character.
> 
> Tracking the number of bytes consumed is easy enough. I like to do in a 
> way that is consistent with D's normal range paradigm.
> 
> Two candidate approaches:
> a) Instead of having the range return the individual values, it could 
> return a tuple containing the value and the number of bytes consumed.
> 
> b) Give the input range an extra member function which returns the 
> number of bytes consumed. The caller could call this after 'empty()' 
> returns true to find the amount of data consumed.
> 
> Both will work, but I'm not especially satisfied with either. Approach 
> (a) seems more consistent with the typical range paradigms, but also 
> more of a hassle for callers.
> 
> Is there a better way to write this?

I had exactly the same problems. I created this to solve the problem, 
I've barely tested it, but I plan to use it with all my parsing 
utilities on iopipe:

https://code.dlang.org/packages/bufref
https://github.com/schveiguy/bufref/blob/master/source/bufref.d

It's not documented fully. But partway down, you will see something 
called `bwin`. This is a range that tries to wrap exactly another range 
that is considered a buffer (random access, also includes narrow 
strings), but keeps track of the location of the data inside the buffer 
(the BufRef). Then, you can access the actual data from the original 
buffer using the BufRef and concrete function.

My plan was to store the BufRef items instead of the actual slices, 
because then they are easy to update if for instance you release the 
front of the buffer, or move the data to the front of the buffer (common 
occurrences in iopipe).

If you have ideas or questions, let me know, possibly on the github 
project for it.

-Steve