Request for comments: std.d.lexer

[Andrei Alexandrescu]

On 2/9/13 3:07 AM, Jonathan M Davis wrote:
> On Friday, February 08, 2013 23:48:50 Walter Bright wrote:
>> On 2/8/2013 12:01 AM, Jonathan M Davis wrote:
>>> It's not quite a use case where
>>> ranges shine - especially when efficiency is a top priority.
>>
>> A more problematic case is dmd's lexer relies on a 0 byte at the end to be a
>> "sentinel" for the end of file. Without such a sentinel, every consumption
>> of a source character requires two operations rather than one.
>
> I didn't know that. That's a cute trick. But yeah, without controlling the
> input, it's not possible, and that won't work with a general implementation in
> a library. It would be trivial enough to put a wrapper around the input to add
> the 0 byte at the end, but the wrapper would still have to keep checking for
> empty, so you wouldn't gain anything.

That's not a problem. You may require that the input has a terminating zero.

Andrei