std.d.lexer: pre-voting review / discussion

[Timon Gehr]

On 09/12/2013 06:40 PM, H. S. Teoh wrote:
>
> I vote for Tok!"..." to denote token types.
>
> Question: what's the implementation of Tok?  Does it fit into an enum?
> What's the underlying representation? I imagine some kind of canonical
> mapping into an integral type would be desired, to maximize runtime
> performance.


This is just a quick hack. One would maybe want to avoid the unwarranted 
copies and linear searches at compile time:

import std.algorithm;

private enum symbols = ["i",".","..",",","0",/+...+/];

struct TokenType{
     private uint _tag; // or smaller type
     private this(int tag){_tag=tag;}
     string toString(){ // (optional)
         static array(R)(R s){ string[] r; foreach(x;s) r~=x; return r; }
         static immutable strs=array(symbols.map!((a)=>`Tok!"`~a~`"`));
         return strs[_tag];
     }
}

template Tok(string name)if(symbols.countUntil(name)!=-1){
     enum Tok=TokenType(cast(uint)symbols.countUntil(name));
}

import std.stdio;
void main(){
     enum x = Tok!"i";
     writeln(x);
     writeln(Tok!"0");
}