Let's stop parser Hell

[Dmitry Olshansky]

On 08-Jul-12 00:50, Roman D. Boiko wrote:
> On Saturday, 7 July 2012 at 20:29:26 UTC, Dmitry Olshansky wrote:
>> And given the backtracking nature of PEGs you'll do your distributed
>> thing many times over or ... spend a lot of RAM to remember not to
>> redo it. I recall lexing takes even more then parsing itself.
>
> I think that your conclusions are about statistical evidences of PEG
> misuses and poor PEG parser implementations. My point was that there is
> nothing fundamentally worse in having lexer integrated with parser, but
> there are performance advantages of having to check less possible cases
> when the structural information is available (so that lexSmth could be
> called when Smth is expected, thus requiring less switch branches if
> switch is used).

You may misunderstood me as well, the point is still the same:
there are 2 things - notation and implementation, the fact that lexer is 
integrated in notation like in PEGs is not related to the fact that PEGs 
in their classic definition never use term Token and do backtracking 
parsing essentially on character level.



> As for lexing multiple times, simply use a free list of terminals (aka
> tokens). I still assume that grammar is properly defined, so that there
> is only one way to split source into tokens.
>

Tokens.. there is no such term in use if we talk about 'pure' PEG.

-- 
Dmitry Olshansky