std.d.lexer: pre-voting review / discussion

[Walter Bright]

On 9/11/2013 10:10 PM, deadalnix wrote:
> See my comment, it is possible, with increased parser complexity, to handle many
> cases where you don't know what you are parsing yet. Doing so, lookahead is only
> required to find matching closing token. I suspect that a fast path in the lexer
> for that precise use case may be faster than buffering tokens, as it allow to
> save one branch per token.

I don't believe that, because you can see about anything for tokens in lookahead 
and so have to duplicate nearly the whole lexer anyway for the 'fast path', but 
you're free to try it out and prove me wrong.