Let's stop parser Hell

[Jonathan M Davis]

On Thursday, July 05, 2012 18:04:11 Roman D. Boiko wrote:
> On Thursday, 5 July 2012 at 15:40:53 UTC, Jacob Carlborg wrote:
> > On 2012-07-05 15:08, Roman D. Boiko wrote:
> >> Anyway I propose to enumerate major use cases first.
> > 
> > Haven't we already done that a couple of times.
> 
> Well, we did something like that for DCT... but I doubt that it
> would fit general needs.
> 
> If we had, why haven't they been analyzed, classified, discussed,
> etc.? Or have they?

It's been discussed before, but there are some obvious use cases such as 
syntax highlighting, code formatting, and manipulation of D source files (e.g. 
to strip out the unit tests).

We need to have a lexer in Phobos which parsers D code and generates a range 
of tokens, and we need to have a parser in Phobos which operates on a range of 
tokens. As discussed previously, there is desire to have the lexer and parser 
ported from dmd's frontend to D for Phobos so that what we get is easily 
maintainable alongside dmd's frontend and produces the same results (and is 
fast). It's _also_ desirable that we get a lexer/parser generator into Phobos 
for generating lexers/parsers for whatever language you might want to generate 
them for. Pegged is a good example of what can be done, and I think that 
Andrei was trying to get Philippe to prepare a submission to Phobos from it 
(though I'm not sure), but regarldess of whether pegged (or anything based on 
it) makes it into Phobos, we definitely want something similar.

So, we have a fair idea of some of the stuff that we want, but it's a question 
of time and effort. I keep intending to port dmd's lexer to D for submission to 
Phobos, but I've been too busy to do it. At least a couple of other people 
have also expressed interest in doing it, but no one has submitted anything 
for Phobos. So, it remains undone, and anything which would need to lex or 
parse D code has to find its own solution. As with most everything around here, 
it's a question of people being having the time and being willing to put in 
the effort to do it. It's all too common for someone to suggest that we should 
do something or implement something without ever attempting to do it 
themselves, and in general, stuff around here gets done because someone really 
wants it done, takes the time to do it, and sees it through until its done and 
in Phobos.

- Jonathan M Davis