Optional semicolons

[Rainer Deyke]

Benji Smith wrote:
> The ECMAScript spec basically says: virtual semicolons must be inserted
> at end-of-line whenever the non-insertion of semicolons would result in
> an erroneous parse.

And I don't like that rule, and I've proposed a sensible alternative, but...

> So there are really only three ways to handle it, and all of them are
> insane:

> 
> 1) Treat the newline character as a token (rather than as skippable
> whitespace) and include that token as an optional construct in every
> single production where it can legally occur. This results in hundreds
> of optional semicolons throughout the grammar, and makes the whole thing
> a nightmare to read, but at least it still uses a one-pass CFG.
> 
>     CLASS :=
>       "class"
>       NEWLINE?
>       IDENTIFIER
>       NEWLINE?
>       "{"
>       NEWLINE?
>       (
>         MEMBER
>         NEWLINE?
>       )*
>       "}"

4) Treat newlines as tokens.  Whenever a non-newline token appears in
the grammar and a newline token appears in the token stream, skip an
arbitrary number of newline tokens to get to the next non-newline token.

This is equivalent to replacing every non-newline terminal symbol X in
the grammar with the (NEWLINE* X) sequence.  You obviously wouldn't do
that by hand.  You've got a general purpose computer in front of you.
Use it!

> The thing about most compiler construction tools is that they don't
> allow interaction between the context-guided tokenization, and they're
> not designed for the creation of backup-and-retry processing, or the
> insertion of virtual tokens into the token stream.

And don't use outdated crappy compiler construction tools.


-- 
Rainer Deyke - rainerd at eldwood.com