Lexer and parser generators using CTFE

[Ben Hanson]

Hi Nick,

On Wednesday, 29 February 2012 at 17:16:44 UTC, Nick Sabalausky
wrote:
> One thing I've been wanting to see in a lexer's regex would be 
> a way to
> match things like D's nested comments or:

I have already implemented a lexer generator that can handle
recursion in the token definitions (using multiple lexer states).
See http://www.benhanson.net/lexertl.html

The library is C++ and generates lexers at runtime, but the
concepts should be easily transferable. Basically I have a
boolean for pop in the end state and a separate column for
push_dfa_:

      if (end_state_)
      {
          // Return longest match
          if (pop_)
          {
              start_state_ = results_.stack.top ().first;
              results_.stack.pop ();
          }
          else if (push_dfa_ != results_.npos ())
          {
              results_.stack.push (typename results::id_type_pair
                  (push_dfa_, id_));
          }
.
.
.

I'm happy to answer any questions etc.

Regards,

Ben