Request for comments: std.d.lexer

[Walter Bright]

On 1/27/2013 2:17 AM, Philippe Sigaud wrote:
> Walter seems to think if a lexer is not able to vomit thousands
> of tokens a seconds, then it's not good.

Speed is critical for a lexer.

This means, for example, you'll need to squeeze pretty much all storage 
allocation out of it. A lexer that does an allocation per token will is not 
going to do very well at all.