Lexer and parser generators using CTFE

[Andrei Alexandrescu]

On 2/28/12 12:46 PM, Martin Nowak wrote:
> I wrote a generic lexer generator some time ago.
> It already let to some compiler O(N^2) optimizations, because the token
> declarations sneak into the mangling :(.
> I also finally added a workaround for a remaining CTFE bug (#6815).
>
> https://gist.github.com/1255439 - lexer generator
> https://gist.github.com/1262321 - complete and fast D lexer

Wow, now we have an embarrassment of riches!

> I've ditched an attempt to write a parser combinator. It was based on
> expression templates and ended up at spirit craziness.

I'm not sure what a parser combinator is. I think grammar inheritance 
could actually be interesting to explore.

> <PERSONAL OPINION
> The hassle of providing good error messages and synthesizing parse results
> in a generic parser outweigh the benefit of a declarative grammar.
> /PERSONAL OPINION>

I think synthesizing ASTs is a solved problem, see my post before. Error 
messages are still a hassle, but I missed the point they became good in 
hand-written parsers :o).

> A lot becomes feasible from the CTFE perspective,
> despite some bugfixes I only miss exp and log currently.
>
> I do not agree that it's the right moment to write a parser though.
> It hits the first of phobos two biggest shortcomings, the lack of a good
> I/O
> system and the missing Allocators.
> Any parser written now will either risk to not play nice with ranges
> or has to come up with it's own buffering again.

Agreed, but that doesn't seem like the largest hurdle to me.


Andrei