Looking for champion - std.lang.d.lex

[Walter Bright]

Andrei Alexandrescu wrote:
> During compilation, such non-tokens are recognized as code by the lexer 
> generator and called appropriately. A comprehensive library of such 
> routines completes a useful library.

I agree, a set of "canned" and heavily optimized lexing functions for common 
things like identifiers, numbers, comments, etc., would make a lexing library 
much more practical.

Those will work great for inventing DSLs, but for existing languages, the 
trouble is that the different languages have subtle variations on how they 
handle them. For example, D's numeric literals allow embedded underscores. Go 
doesn't overflow on numeric literals. Javascript has some wacky rules to 
distinguish a comment from a regex. The \uNNNN letters allowed in identifiers in 
some languages.

So while a general purpose lexing library will be very useful, for lexing D code 
(and Java, Javascript, etc.) a custom one will probably be much more practical.