std.d.lexer: pre-voting review / discussion

[dennis luehring]

Am 12.09.2013 10:15, schrieb Brian Schott:
> I got some time to work on the lexer this evening. Changeset
> here:
> https://github.com/Hackerpilot/phobos/commit/9bdb7f97bb8021f3b0d0291896b8fe21a6fead23#std/d/lexer.d
>
> The DDoc page has moved here:
> http://hackerpilot.github.io/experimental/std_lexer/phobos/std_d_lexer.html
>
> * There are a few more unit tests now
> * bitAnd renamed to amp
> * slice rename to dotdot
> * Much more cross-referencing in the doc comments
> * Start line and column can be specified in the lexer config
>

problem: many occurences of the same string

you should use constants for the tokens (and others)

string asm_token = "asm";
...

immutable(string[TokenType.max + 1]) tokenValues = [
   ...
   asm_token
   ...
]

and reuse these constants in your "optimization"

maybe you can replace these lines with something getting feed with 
asm_token and give the same result but without 'a' and "sm" as splitted 
and different magic values - maybe a nice template or subrange...

case 'a': if (input[1..$].equal("sm")) return TokenType.asm_; else
...
break;

and in your unit tests

for example

on auto expected =