Let's stop parser Hell

[Jonathan M Davis]

On Sunday, July 08, 2012 14:13:18 Jacob Carlborg wrote:
> On 2012-07-07 19:53, Jonathan M Davis wrote:
> > There are multiple issues here. The one that Andrei is raising is the fact
> > that D isn't properly formalized. Essentially, the compiler _is_ the spec,
> > and while the online spec _mostly_ follows it, it doesn't entirely, and
> > the online spec isn't always as precise as it needs to be regardless.
> > With a fully formalized spec, it should be possible to fully implement a
> > D compiler from the spec alone, and I don't think that that's currently
> > possible.
> 
> That's been a problem for a long time.

True, but Andrei is basically arguing that porting dmd's lexer and parser to D 
would reduce the incentive to fix this, whereas having a new lexer/parser would 
encourage fixing it (though he's arguing for purely using a generative parser 
rather than writing a D-specific one).

> > IMHO, the reason that porting dmd's lexer and parser would be of great
> > benefit is primarily maintenance. It makes it much easier to keep Phobos'
> > lexer and parser in line with dmd, making discrepencies less likely, but
> > it arguably reduces the incentive to improve the spec.
> 
> But then it sounds like the best solution would be not to have a
> lexer/parser based on DMD and instead making sure the spec is correct.

That would be the question. Given enough time, what I'd probably want to do 
would be to port dmd's lexer and parser to D in order to properly figure it all 
out, updating the spec in the process, and _then_ go write one from scratch, 
possibly basing some of it on how dmd did it, possibly not. But I don't really 
have the time for any of that right now, and most of the focus right now from 
people interested in parsing seems to be on pegged and parser generators 
(which are very cool and in some ways much more interesting, but I seriously 
question that that's the performant way to go if you're looking to parse D 
specifically). So, who knows what we're going to get out of this and when we'll 
get it.

> What keeps popping up in my head is the scenario where users are
> complaining about the frontend in Phobos not behaving as their compiler.
> If this is due to they are out of sync, bugs or not following the spec
> doesn't matter.
> 
> I still thinks D is changing too much to have two separate
> implementations of the compiler and a library like this.

Well, for the lexer and parser, we're probably okay (or very close to it). As 
Daniel pointed out elsewhere in this thread, that part of the frontend doesn't 
change much (particularly the lexer). There's definitely still some churn, but 
it's nothing like it used to be.

- Jonathan M Davis