std.d.lexer : voting thread

[deadalnix]

On Wednesday, 9 October 2013 at 04:38:02 UTC, Andrei Alexandrescu 
wrote:
> On 10/8/13 9:32 PM, deadalnix wrote:
>> Overall, I think this is going into the right direction. 
>> However, there
>> is one thing I don't like with that design.
>>
>> When you go throw the big switch of death, you match the 
>> beginning of
>> the string and then you go back to a function that will test 
>> where does
>> it come from and act accordingly. That is kind of wasteful.
>>
>> What SDC does is that it calls a function-template with the 
>> part matched
>> by the big switch of death passed as template argument. The 
>> nice thing
>> about it is that it is easy to trnsform this compile time 
>> argument into
>> a runtime one by simply forwarding it (what is done to parse 
>> identifier
>> that begins by a keyword for instance).
>
> I think a bit of code would make all that much clearer.
>
> Andrei

Sure.

So here is the lexer generation infos (this can be simplified by 
using the tok!"foobar" thing) : http://dpaste.dzfl.pl/7ec225ee

Using theses infos, a huge switch based boilerplate is generated. 
Each "leaf" of the huge switch tree call a function template as 
follow, by passing as template argument what has been matched so 
far. You can then proceed as follow :
http://dpaste.dzfl.pl/f2f0d22c

You may wonder about the "?lexComment". The boilerplate generator 
understand ? as an indication that lexComment may or may not 
return a token (depending on lexer configuration) and generate 
what is needed to handle that (by testing if the function return 
a token, via some static ifs).

You obviously ends up with a log of instance of 
lexIdentifier(string s)(), but this simply forward to 
lexIdentifier()(string s) and the forwarding function is removed 
trivially by the inliner.