Lexers (again)

[Brian Schott]

On Friday, 13 December 2013 at 10:17:49 UTC, Brian Schott wrote:
> I've been working on the next attepmpt at a std.lexer / 
> std.d.lexer recently. You can follow the progress on Github 
> here: https://github.com/Hackerpilot/lexer-work

I've ported DScanner over to this new lexer code. It's on a 
branch here: 
https://github.com/Hackerpilot/Dscanner/tree/NewLexer.

One limitation I've noticed with the new tok!"tokenName" approach 
is that while dmd has no problem with

case tok!"class":

it does have a problem with

goto case tok!"class":

I managed to work around this by adding new labels and "goto"-ing 
them instead. Is this a bug or intentional?