Request for comments: std.d.lexer

[Walter Bright]

On 1/27/2013 12:18 PM, Philippe Sigaud wrote:
>> I concur. One of the biggest reason* there is a separate lexer step is
>> because it could be made to do this stage very-very fast. Then the rest of
>> the parser will greatly benefit from this underlying speed.
>>
>> *Otherwise we could have just as well add the lexer stage as simple rules to
>> the grammar that treats all of codepoints as terminals.
>
> ... which is exactly what parsing expression grammars (and other
> scannerless parsers) do.
>
> AFAICT, one interesting consequence is the ability to have composition
> of grammars, which I sure have with Pegged. But maybe it's not related
> that much, that's not something I stopped to think about.
> In any case, grammar composition is something I've learnt to like quite a lot.
>

If you do such a thing when designing a language, I guarantee that you will find 
it impossible to resist having custom "tokens" in various parts of the grammar. 
This will make it impossible to write tools that only need to tokens, such as 
those syntax highlighting editors that really are token highlighters.