Sentinel InputRange ?

[Walter Bright]

On 2/9/2013 6:37 AM, Andrei Alexandrescu wrote:
> On 2/9/13 3:07 AM, Jonathan M Davis wrote:
>> On Friday, February 08, 2013 23:48:50 Walter Bright wrote:
>>> On 2/8/2013 12:01 AM, Jonathan M Davis wrote:
>>>> It's not quite a use case where
>>>> ranges shine - especially when efficiency is a top priority.
>>>
>>> A more problematic case is dmd's lexer relies on a 0 byte at the end to be a
>>> "sentinel" for the end of file. Without such a sentinel, every consumption
>>> of a source character requires two operations rather than one.
>>
>> I didn't know that. That's a cute trick. But yeah, without controlling the
>> input, it's not possible, and that won't work with a general implementation in
>> a library. It would be trivial enough to put a wrapper around the input to add
>> the 0 byte at the end, but the wrapper would still have to keep checking for
>> empty, so you wouldn't gain anything.
>
> That's not a problem. You may require that the input has a terminating zero.

Perhaps we can formalize this a bit. Define a SentinelInputRange, which has an 
additional manifest constant with the name 'sentinel'. empty becomes defined as:

     empty = front == sentinel;

popFront() does not advance if empty. Advancing over a SentinelInputRange can be 
done the usual:

     for (r = range; !r.empty; r.popFront()) { auto e = r.front; }

or can be done as:

     for (r = range; r.front != sentinel; r.popFront()) { auto e = r.front; }

This should work well with 0 terminated strings. The output of the lexer should 
also be a SentinelInputRange, with TOKeof as the sentinel.