std.d.lexer : voting thread

[Andrei Alexandrescu]

On 10/8/13 7:02 AM, Jakob Ovrum wrote:
> On Tuesday, 8 October 2013 at 04:37:31 UTC, Andrei Alexandrescu wrote:
>> So I guess it's your turn.
>
> I was going to cook something up with `groupBy` (taken from the
> still-open Phobos PR #1186) and `toTypeTuple`(from Phobos PR #1472, also
> still open!), but the former isn't CTFEable. Blergh. I'm still adamant
> this is the way to go, but I'm putting away the torch for now.

Fair enough. (Again, it would be unfair to compare an existing design 
against a hypothetical one.) I suspect at some point you will need to 
generate some custom code, which will come as a string that you need to 
mixin.

But no matter. My most significant bit is, we need a trie lexer 
generator ONLY from the token strings, no TK_XXX user-provided symbols 
necessary. If all we need is one language (D) this is a non-issue 
because the library writer provides the token definitions. If we need to 
support user-provided languages, having the library manage the string -> 
small integer mapping becomes essential.


Andrei