Lexer and parser generators using CTFE

[H. S. Teoh]

On Tue, Feb 28, 2012 at 07:46:04PM +0100, Martin Nowak wrote:
[...]
> I wrote a generic lexer generator some time ago.
> It already let to some compiler O(N^2) optimizations, because the token
> declarations sneak into the mangling :(.
> I also finally added a workaround for a remaining CTFE bug (#6815).
> 
> https://gist.github.com/1255439 - lexer generator
> https://gist.github.com/1262321 - complete and fast D lexer

Cool! I'll have to take a look at this sometime.


[...]
> <PERSONAL OPINION
> The hassle of providing good error messages and synthesizing parse results
> in a generic parser outweigh the benefit of a declarative grammar.
> /PERSONAL OPINION>

But things like lex/yacc have been useful throughout the years. With D's
delegates, lexer/parser action rules should be even cleaner, no?


> A lot becomes feasible from the CTFE perspective, despite some
> bugfixes I only miss exp and log currently.

All of std.math should have CTFE versions so that we can perform complex
calculations at compile-time (e.g., create table of scaled sin/cos
values for use in high-performance 3D apps -- no need to waste time to
generate these tables at startup time).


> I do not agree that it's the right moment to write a parser though.
> It hits the first of phobos two biggest shortcomings, the lack of a
> good I/O system and the missing Allocators.  Any parser written now
> will either risk to not play nice with ranges or has to come up with
> it's own buffering again.

OTOH, perhaps once we start writing a lexer/parser generator, that will
force us to fix the I/O and allocator system. Without a good project to
motivate us to fix these tedious issues, it seems that we just lose
inspiration to do it after a while.


T

-- 
The computer is only a tool. Unfortunately, so is the user. -- Armaphine, K5