How do I write a "lexer or parser generators"

[Timon Gehr]

On 01/23/2014 09:12 AM, OP wrote:
> Taking the syntax below I'm not sure how to fork a state and discard the
> invalid one.
>
>> foo[5] = var2
>> foo[5] foo = var2
>
> Here when I see foo[5] I'm either accessing an array (first statement)
> or declaring variables as an array of 5 elements (second statement).
> Just like `Foo&foo` could be a reference or could be an AND statement.
> Foo is definitely processed on its own, I don't know how to process it
> as both (a fork) and continue on the parser to find a valid path.

A simple way to deal with this is to determine whether the parser is at 
the beginning of a declaration or an expression statement, eg. save the 
current position in the input stream, and try to skip (parse without 
building an AST) something looking like a declaration, remember if it 
worked and reset the position. If this succeeds, parse the declaration, 
otherwise parse an expression statement.