std.d.lexer: pre-voting review / discussion

[Walter Bright]

On 9/11/2013 3:09 PM, Michel Fortin wrote:
> On 2013-09-11 18:49:44 +0000, Walter Bright <newshound2 at digitalmars.com> said:
>
>> 6. No clue how lookahead works with this. Parsing D requires arbitrary lookahead.
>
> Since this is a forward range, you can save() it at one location and continue
> parsing, and then restart parsing from the savepoint.
> https://github.com/Hackerpilot/phobos/blob/master/std/d/lexer.d#L2324
>
> This also means that the underlying range used as input to the lexer must be a
> forward range that you can save() too.

That's correct, but that implies re-lexing the tokens, which has negative 
performance implications.