std.d.lexer : voting thread

[Andrei Alexandrescu]

On 10/7/13 5:16 PM, Andrei Alexandrescu wrote:
> On 10/4/13 5:24 PM, Andrei Alexandrescu wrote:
>> On 10/2/13 7:41 AM, Dicebot wrote:
>>> After brief discussion with Brian and gathering data from the review
>>> thread, I have decided to start voting for `std.d.lexer` inclusion into
>>> Phobos.
>>
>> Thanks all involved for the work, first of all Brian.
>>
>> I have the proverbial good news and bad news. The only bad news is that
>> I'm voting "no" on this proposal.
>>
>> But there's plenty of good news.
>>
>> 1. I am not attempting to veto this, so just consider it a normal vote
>> when tallying.
>>
>> 2. I do vote for inclusion in the /etc/ package for the time being.
>>
>> 3. The work is good and the code valuable, so even in the case my
>> suggestions (below) will be followed, a virtually all code pulp that
>> gets work done can be reused.
> [snip]
>
> To put my money where my mouth is, I have a proof-of-concept tokenizer
> for C++ in working state.
>
> http://dpaste.dzfl.pl/d07dd46d

I made an improvement to the way tokens are handled. In the paste above, 
"tk" is a function. A CTFE-able function that just returns a 
compile-time constant, but a function nevertheless.

To actually reduce "tk" to a compile-time constant in all cases, I 
changed it as follows:

   template tk(string symbol) {
     import std.range;
     static if (symbol == "") {
       // Token ID 0 is reserved for "unrecognized token".
       enum tk = TokenType2(0);
     } else static if (symbol == "\0") {
       // Token ID max is reserved for "end of input".
         enum tk = TokenType2(
           cast(TokenIDRep) (1 + tokens.length + reservedTokens.length));
     } else {
         //enum id = chain(tokens, reservedTokens).countUntil(symbol);
       // Find the id within the regular tokens realm
       enum idTokens = tokens.countUntil(symbol);
       static if (idTokens >= 0) {
         // Found, regular token. Add 1 because 0 is reserved.
         enum id = idTokens + 1;
       } else {
         // not found, only chance is within the reserved tokens realm
         enum idResTokens = reservedTokens.countUntil(symbol);
         enum id = idResTokens >= 0 ? tokens.length + idResTokens + 1 : -1;
       }
       static assert(id >= 0 && id < TokenIDRep.max,
                     "Invalid token: " ~ symbol);
       enum tk = TokenType2(id);
     }

This is even better now because token types are simple static constants.


Andrei