Request for comments: std.d.lexer

[Philippe Sigaud]

> I concur. One of the biggest reason* there is a separate lexer step is
> because it could be made to do this stage very-very fast. Then the rest of
> the parser will greatly benefit from this underlying speed.
>
> *Otherwise we could have just as well add the lexer stage as simple rules to
> the grammar that treats all of codepoints as terminals.

... which is exactly what parsing expression grammars (and other
scannerless parsers) do.

AFAICT, one interesting consequence is the ability to have composition
of grammars, which I sure have with Pegged. But maybe it's not related
that much, that's not something I stopped to think about.
In any case, grammar composition is something I've learnt to like quite a lot.