Writing a JFlex lexer for D - have an issue with cycles

[FatalCatharsis via Digitalmars-d]

On Sunday, 22 January 2017 at 23:20:27 UTC, Basile B. wrote:
>
> One way to do this is not to do anything special for q{. Just 
> add a token for q{ and continue normal lexing. The token string 
> content must be valid tokens so it should work.
>
> In facts it depends on what the scanner just be used for...for 
> serious compiler things this is not acceptable. For 
> highlighting or tools this might be okay. The real problem is 
> that a token string should be printable... so it exists as a 
> whole before being tokenized.

This is for tooling, but I still want it to be accurate. Are you 
suggesting I could move handling token string to the syntax side 
of things? I could remove it from the lexer entirely and do like 
you said, have a token g{ and a syntax production that uses it 
like so:

      Token ::=
          IDENTIFIER |
          STRING_LITERAL |
          TokenString |
          CHARACTER_LITERAL |
          INTEGER_LITERAL |
          FLOAT_LITERAL |
          KEYWORD |
          OPERATOR

      TokenString ::= TOKEN_STRING_OP Token RIGHT_CURLY_OP

and then anywhere where there is a StringLiteral (e.g. 
StringLiterals, AsmInstruction, TemplateSingleArgument), then 
provide an equivalent production that uses TokenString in place 
of StringLiteral. Would this be equivalent? (Just want another 
set of eyes before I call it good.)