std.d.lexer requirements

[Walter Bright]

On 8/1/2012 9:41 PM, H. S. Teoh wrote:
> Whether it's part of the range type or a separate lexer type,
> *definitely* make it possible to have multiple instances. One of the
> biggest flaws of otherwise-good lexer generators like lex and flex
> (C/C++) is that the core code assumes a single instance, and
> multi-instances were glued on after the fact, making it a royal pain to
> work with anything that needs lexing multiple things at the same time.

Yup. I keep trying to think of a way to lex multiple files at the same time in 
separate threads, but the problem is serializing access to the identifier table 
will likely kill off any perf gain.