Request for comments: std.d.lexer

[Andrei Alexandrescu]

On 2/9/13 10:05 AM, Jacob Carlborg wrote:
> On 2013-02-09 09:05, Jonathan M Davis wrote:
>
>> The lexer that I'm writing doesn't even support pure input ranges,
>> because it
>> has to be able to save to do what it does, and the only reason that
>> decodeFront exists is because I wrote it to support that use case. And
>> needing
>> to operate on ranges of code units is likely to be quite rare, and the
>> function is quite new, so I don't expect that much code is affected by
>> any bugs
>> in decodeFront or that much would be broken were it to be changed.
>
> Following this discussion it seems it's complicated to support ranges in
> a lexer and still maintain the speed. Is it really worth the trouble to
> support ranges?

Requiring a random-access range of ubyte with a terminating zero may be 
the most general approach to a fast lexer - and that's fine.

Andrei