Proposal for SentinelInputRange

[Timon Gehr]

On 02/28/2013 05:48 PM, Dmitry Olshansky wrote:
> ...
>
> line 300:
>      case 0x1A:          // ^Z means end of file
>      case 0:
>                          break;
>
> On the lines you noted it claimed that that 0x1a is outdate. Along with
> the fact that you allocate filesize+2 and fill the last 2 bytes with zeros.
>
> In any case I see 0 and 0x1a as 2 values that act like sentinels i.e. a
> tuple. And this is what spec says - any one of them is a sentinel.
> Correct me if I'm wrong.
>

A sentinel is some data the original data is augmented with in order to 
simplify its processing.

The lexer acts the same on 0x1A and 0, but only the additional 0 at the 
end which does not occur in the input is the sentinel. The lexer may 
even encounter a 0 that is not a sentinel.