std.d.lexer : voting thread

[Walter Bright]

On 10/4/2013 5:24 PM, Andrei Alexandrescu wrote:
>  [...]

Some points:

1. This is a replacement for the switch statement starting at around line 505 in 
advance()
 
https://github.com/Hackerpilot/phobos/blob/9bdb7f97bb8021f3b0d0291896b8fe21a6fead23/std/d/lexer.d
It is not a replacement for the rest of the lexer.

2. Instead of explicit token type enums, such as:

     mod, /// %

it would just be referred to as:

     tok!"%"

Andrei pointed out to me that he has fixed the latter so it resolves to a small 
integer - meaning it works efficiently as cases in switch statements. This 
removes my primary objection to it.

3. This level of abstraction combined with efficient generation cannot be 
currently done in any other language. Hence, it makes for a sweet showcase of 
what D can do.

Hence, I think we ought to adapt Brian's lexer by replacing the switch with 
Andrei's trie searcher, and replacing the enum TokenType with the tok!"string" 
syntax.