Looking for champion - std.lang.d.lex

[Andrei Alexandrescu]

On 10/22/10 16:28 CDT, Sean Kelly wrote:
> Andrei Alexandrescu Wrote:
>>
>> I have in mind the entire implementation of a simple design, but never
>> had the time to execute on it. The tokenizer would work like this:
>>
>> alias Lexer!(
>>       "+", "PLUS",
>>       "-", "MINUS",
>>       "+=", "PLUS_EQ",
>>       ...
>>       "if", "IF",
>>       "else", "ELSE"
>>       ...
>> ) DLexer;
>>
>> Such a declaration generates numeric values DLexer.PLUS etc. and
>> generates an efficient code that extracts a stream of tokens from a
>> stream of text. Each token in the token stream has the ID and the text.
>
> What about, say, floating-point literals?  It seems like the first element of a pair might have to be a regex pattern.


Yah, with regard to such regular patterns (strings, comments, numbers, 
identifiers) there are at least two possibilities that I see:

1. Go the full route of allowing regexen in the definition. This is very 
hard because you need to generate an efficient (N|D)FA during compilation.

2. Pragmatically allow "fallthrough" routines, i.e. if nothing in the 
compile-time table matches, just call onUnrecognizedString(). In 
conjunction with a few simple specialized functions, that makes it very 
simple to define arbitrarily complex lexers where the bulk of the work 
(and the most tedious part) is done by the D compiler.


Andrei