Official D Grammar

[Walter Bright]

On 4/1/2013 4:18 PM, Brian Schott wrote:
> I've pretty much finished up my work on the std.d.lexer module. I am waiting for
> the review queue to make some progress on the other (three?) modules being
> reviewed before starting a thread on it.
>
> In the meantime I've started some work on an AST module for Phobos that contains
> the data types necessary to build up a parser module so that we can have a
> standard set of code build D dev tools off of. I decided to work directly from
> the standard on dlang.org for this to make sure that my module is correct and
> that the standard is actually correct.
>
> I've seen several threads on this newsgroup complaining about the state of the
> standard and unfortunately this will be another one.
>
> 1) Grammar defined in terms of things that aren't tokens. Take, for example,
> PropertyDeclaration. It's defined as an "@" token followed by... what? "safe"?
> It's not a real token. It's an identifier. You can't parse this based on
> checking the token type. You have to check the type and the value.

True, do you have a suggestion?

>
> 2) Grammar references rules that don't exist. UserDefinedAttribute is defined in
> terms of CallExpression, but CallExpression doesn't exist elsewhere in the
> grammar. BaseInterfaceList is defined in terms of InterfaceClasses, but that
> rule is never defined.

Yes, this needs to be fixed.

>
> 3) Unnecessary rules. KeyExpression, ValueExpression, ScopeBlockStatement,
> DeclarationStatement, ThenStatement, ElseStatement, Test, Increment, Aggregate,
> LwrExpression, UprExpression, FirstExp, LastExp, StructAllocator,
> StructDeallocator, EnumTag, EnumBaseType, EmptyEnumBody, ConstraintExpression,
> MixinIdentifier, etc... are all defined in terms of only one other rule.

Using these makes documentation easier, and I don't think it harms anything.


> I think that we need to be able to create a grammar description that:
> * Fits in to a single file, so that a tool implementer does not need to collect
> bits of the grammar from the various pages on dlang.org.
> * Can be verified to be correct by an existing tool such as Bison, Goldie,
> JavaCC, <your favorite here> with a small number of changes.
> * Is part of the dmd/dlang repositories on github and gets updated every time
> the language changes.
>
> I'm willing to work on this if there's a good chance it will actually be
> implemented. Thoughts?

I suggest doing this as a sequence of pull requests, not doing just one big one.