Lexer and parser generators using CTFE

[H. S. Teoh]

On Tue, Feb 28, 2012 at 01:59:21AM -0600, Andrei Alexandrescu wrote:
> I'm starting a new thread on this because I think the matter is of
> strategic importance.

+1.


> We all felt for a long time that there's a lot of potential in CTFE,
> and potential applications have been discussed more than a few times,
> ranging from formatting strings parsed to DSLs and parser
> generators.

CTFE is one of the big features that attracted me to D.


[...]
> This is the kind of stuff I've had an eye on for the longest time.
> I'm saying it's of strategic importance because CTFE technology,
> though not new and already available with some languages, has unique
> powers when combined with other features of D. With CTFE we get to do
> things that are quite literally impossible to do in other languages.

For example?


> We need to have a easy-to-use, complete, seamless, and efficient
> lexer-parser generator combo in Phobos, pronto. The lexer itself could
> use a character-level PEG or a classic automaton, and emit tokens for
> consumption by a parser generator. The two should work in perfect
> tandem (no need for glue code). At the end of the day, defining a
> complete lexer+parser combo for a language should be just a few lines
> longer than the textual representation of the grammar itself.

Definitely agree. I frequently work with code that requires some amount
of lexing/parsing; having something like this in Phobos would be
absolutely awesome.


> What do you all think? Let's get this project off the ground!
[...]

+1.

On that note, I'd like to mention that CTFE currently has some parts
that need more work:

(1) We need compile-time alternatives to the functions in std.math so
that functions that are currently only implemented in asm can be used in
CTFE.

(2) It should be possible to generate AA literals from CTFE and use them
to initialize things like parsing/lexing lookup tables, etc.. While this
in itself is not a major critical issue, it would make D look bad if we
tout D's CTFE capabilities yet at the same time AA's have to be created
by static this() at runtime.


T

-- 
Which is worse: ignorance or apathy? Who knows? Who cares? -- Erich Schubert