Request for comments: std.d.lexer

[Dmitry Olshansky]

28-Jan-2013 00:18, Philippe Sigaud пишет:
>> I concur. One of the biggest reason* there is a separate lexer step is
>> because it could be made to do this stage very-very fast. Then the rest of
>> the parser will greatly benefit from this underlying speed.
>>
>> *Otherwise we could have just as well add the lexer stage as simple rules to
>> the grammar that treats all of codepoints as terminals.
>
> ... which is exactly what parsing expression grammars (and other
> scannerless parsers) do.
>

That is only a happenstance and is highly overrated. What does it buy 
you is the right question to ask. After all every LL-parser can be 
written avoiding notion of lexer and strangely I see no fuss about it.
So is it just hype? I dunno.

> AFAICT, one interesting consequence is the ability to have composition
> of grammars, which I sure have with Pegged. But maybe it's not related
> that much, that's not something I stopped to think about.
> In any case, grammar composition is something I've learnt to like quite a lot.
>

You can still have composability of grammars. In fact I'd define a 
couple of a less powerful but better optimized variations if I were you.

PEG-style memoization is still possible as long as underlying grammars 
are context-free all the useful usually are.

LR hardly composable ( I haven't seen a single one that does).
But all top-down one always are unless there is some explicit work spent 
to undermine this ability ;)

-- 
Dmitry Olshansky