std.d.lexer: pre-voting review / discussion

[Martin Nowak]

On 09/12/2013 08:17 AM, Walter Bright wrote:
> I don't believe that, because you can see about anything for tokens in
> lookahead and so have to duplicate nearly the whole lexer anyway for the
> 'fast path', but you're free to try it out and prove me wrong.

Me neither and if you have to duplicate the code for this it's a double 
loss.
I think it would be more fruitful to batch lexing, i.e. fill a hole 
memory page (4Kb) with tokens the go back to parsing until you need more.