Lexers (again)

[Andrei Alexandrescu]

On 12/15/13 3:45 AM, Timon Gehr wrote:
> On 12/15/2013 12:12 PM, Brian Schott wrote:
>> On Friday, 13 December 2013 at 10:17:49 UTC, Brian Schott wrote:
>>> I've been working on the next attepmpt at a std.lexer / std.d.lexer
>>> recently. You can follow the progress on Github here:
>>> https://github.com/Hackerpilot/lexer-work
>>
>> I've ported DScanner over to this new lexer code. It's on a branch here:
>> https://github.com/Hackerpilot/Dscanner/tree/NewLexer.
>>
>> One limitation I've noticed with the new tok!"tokenName" approach is
>> that while dmd has no problem with
>>
>> case tok!"class":
>>
>> it does have a problem with
>>
>> goto case tok!"class":
>>
>> I managed to work around this by adding new labels and "goto"-ing them
>> instead. Is this a bug or intentional?
>
> I cannot reproduce your problem. If this does not work, it is a bug.

The problem is that tok is a dynamic value. It should be a static value. 
Current code:

static @property IDType tok(string symbol)()
{
     ...
}

It should be:

template IDType tok(string symbol)()
{
     alias tok = ...;
}

This is important - if the compiler thinks tok is a dynamic value, it'll 
generate crappy switch statements.

BTW @Brian - I didn't look at this in depth yet but it's very promising 
work. Thanks!


Andrei