Official D Grammar

[Brian Schott]

I've pretty much finished up my work on the std.d.lexer module. I 
am waiting for the review queue to make some progress on the 
other (three?) modules being reviewed before starting a thread on 
it.

In the meantime I've started some work on an AST module for 
Phobos that contains the data types necessary to build up a 
parser module so that we can have a standard set of code build D 
dev tools off of. I decided to work directly from the standard on 
dlang.org for this to make sure that my module is correct and 
that the standard is actually correct.

I've seen several threads on this newsgroup complaining about the 
state of the standard and unfortunately this will be another one.

1) Grammar defined in terms of things that aren't tokens. Take, 
for example, PropertyDeclaration. It's defined as an "@" token 
followed by... what? "safe"? It's not a real token. It's an 
identifier. You can't parse this based on checking the token 
type. You have to check the type and the value.

2) Grammar references rules that don't exist. 
UserDefinedAttribute is defined in terms of CallExpression, but 
CallExpression doesn't exist elsewhere in the grammar. 
BaseInterfaceList is defined in terms of InterfaceClasses, but 
that rule is never defined.

3) Unnecessary rules. KeyExpression, ValueExpression, 
ScopeBlockStatement, DeclarationStatement, ThenStatement, 
ElseStatement, Test, Increment, Aggregate, LwrExpression, 
UprExpression, FirstExp, LastExp, StructAllocator, 
StructDeallocator, EnumTag, EnumBaseType, EmptyEnumBody, 
ConstraintExpression, MixinIdentifier, etc... are all defined in 
terms of only one other rule.

I think that we need to be able to create a grammar description 
that:
* Fits in to a single file, so that a tool implementer does not 
need to collect bits of the grammar from the various pages on 
dlang.org.
* Can be verified to be correct by an existing tool such as 
Bison, Goldie, JavaCC, <your favorite here> with a small number 
of changes.
* Is part of the dmd/dlang repositories on github and gets 
updated every time the language changes.

I'm willing to work on this if there's a good chance it will 
actually be implemented. Thoughts?