Writing a JFlex lexer for D - have an issue with cycles

[Basile B. via Digitalmars-d]

On Monday, 23 January 2017 at 00:03:44 UTC, FatalCatharsis wrote:
> On Sunday, 22 January 2017 at 23:20:27 UTC, Basile B. wrote:
>>
>> One way to do this is not to do anything special for q{. [...]
>
> This is for tooling, but I still want it to be accurate. Are 
> you suggesting I could move handling token string to the syntax 
> side of things? I could remove it from the lexer entirely and 
> do like you said, have a token g{ and a syntax production that 
> uses it like so:
>
>      Token ::=
>          IDENTIFIER |
>          STRING_LITERAL |
>          TokenString |
>          CHARACTER_LITERAL |
>          INTEGER_LITERAL |
>          FLOAT_LITERAL |
>          KEYWORD |
>          OPERATOR
>
>      TokenString ::= TOKEN_STRING_OP Token RIGHT_CURLY_OP
>
> [...]

Yes this was the idea but "Profile Anaysis" first answer is 
better. You really should have a single string for any token 
string, even if it's nested. Until the token string is used it 
doesn't need to be tokenized.

The solution that's to tokenize the string is only useful for 
highlighting.