Let's stop parser Hell

[Jacob Carlborg]

On 2012-07-07 19:53, Jonathan M Davis wrote:

> There are multiple issues here. The one that Andrei is raising is the fact
> that D isn't properly formalized. Essentially, the compiler _is_ the spec, and
> while the online spec _mostly_ follows it, it doesn't entirely, and the online
> spec isn't always as precise as it needs to be regardless. With a fully
> formalized spec, it should be possible to fully implement a D compiler from
> the spec alone, and I don't think that that's currently possible.

That's been a problem for a long time.

> Writing a D lexer and parser (if not a full-blown compiler) from scratch would
> help highlight the places in the spec which are lacking, and having it be part
> of Phobos would arguably increase Walter's incentive to make sure that the
> spec is in line with the compiler (and vice versa) so that stuff _other_ than
> the compiler which is based on the spec would be able to match the compiler.
>
> Clang is in a _completely_ different situation. It's a C/C++ compiler, and both
> C and C++ already have official, formalized specs which Clang conforms to (or is
> supposed to anyway). Clang has no control over the spec at all. It merely
> implements it. So, there is no issue of trying to keep other tools or
> compilers in line with Clang due to it being the language's spec like we
> effectively have with dmd. It may help the tools which use Clang to be fully in
> line with Clang and not have to worry about whether Clang implements the spec
> slightly differently, but in theory, if they all follow the spec correctly,
> that isn't in issue (though obviously in practice it can be).

That should be true for D as well when the spec is complete.

> In D's case, all of the major D compilers use the same frontend, which helps
> compatability but harms the spec, because there's less incentive to keep it
> precise and  up-to-date. So, from the perspective of the spec, implementing
> the D lexer and parser for Phobos separately from dmd would be of great
> benefit.

That might be true.

> IMHO, the reason that porting dmd's lexer and parser would be of great benefit
> is primarily maintenance. It makes it much easier to keep Phobos' lexer and
> parser in line with dmd, making discrepencies less likely, but it arguably
> reduces the incentive to improve the spec.

But then it sounds like the best solution would be not to have a 
lexer/parser based on DMD and instead making sure the spec is correct.

> The benefits of having a lexer and parser as a library (regardless of whether
> it's from scratch or a port from dmd) are primarly derived from the fact that
> it makes it much easier to create tools which use them. Such tools no longer
> have to write their own lexers or parsers.

Of course, that's the whole point.

> If the compiler uses the same library, it has the added benefit of making it so
> that any tool using the library will be in sync with the compiler, but if the
> spec is properly formalized and up-to-date, and the library is kep up-to-date
> with _it_, that shouldn't really be a problem. You still have the debate as to
> whether it's better to have a separate implementation based on the spec
> (thereby making it more likely that the spec is correct) or whether it's
> better to have the compiler share the implementation so that the library is
> guaranteed to match the compiler (though not necessarily the spec), but I
> think that that's a separate debate from whether we should have the lexer and
> parser as a library.

What keeps popping up in my head is the scenario where users are 
complaining about the frontend in Phobos not behaving as their compiler. 
If this is due to they are out of sync, bugs or not following the spec 
doesn't matter.

I still thinks D is changing too much to have two separate 
implementations of the compiler and a library like this.

> In all honesty though, I would be surprised if you could convince Walter to
> switch dmd's frontend to Phobos' lexer and parser even once they've been
> implemented. So, while I agree that there are benefits in doing so, I'm not
> sure how much chance you have of ever getting any traction with that.

That is perhaps true.

> Another thing to consider is that that might make it so that gdc and ldc
> couldn't share the same frontend with dmd (assuming that they have to keep
> their frontends in C or C++ -  I don't know if they do) - but if so, that
> would increase the incentive for the spec to be correct if dmd ever started
> using a D frontend.

-- 
/Jacob Carlborg