Looking for champion - std.lang.d.lex

[Sean Kelly]

Andrei Alexandrescu Wrote:
> 
> I have in mind the entire implementation of a simple design, but never 
> had the time to execute on it. The tokenizer would work like this:
> 
> alias Lexer!(
>      "+", "PLUS",
>      "-", "MINUS",
>      "+=", "PLUS_EQ",
>      ...
>      "if", "IF",
>      "else", "ELSE"
>      ...
> ) DLexer;
> 
> Such a declaration generates numeric values DLexer.PLUS etc. and 
> generates an efficient code that extracts a stream of tokens from a 
> stream of text. Each token in the token stream has the ID and the text.

What about, say, floating-point literals?  It seems like the first element of a pair might have to be a regex pattern.