DMD 1.021 and 2.004 releases

[Jascha Wetzel]

Jari-Matti Mäkelä wrote:
> Jascha Wetzel wrote:
> 
>> D's delimited strings can (luckily) be scanned with regular languages,
>> because the enclosing double quotes are required. else the lexical
>> structure wouldn't even be context free and a nightmare for
>> automatically generated lexers.
> 
> Right, thanks.
> 
>> therefore you can match q"[^"]*" and check the delimiters during
>> (context sensitive) semantic analysis.
> 
> But e.g. syntax highlighting needs the semantic info to change the style of
> the text within the delimiters. The analyser also needs to check whether
> the two delimiters match. Like I said above, if the tool doesn't provide
> enough support, you're stuck. I haven't searched for all corner cases, but
> wasn't the old grammar scannable and highlightable with plain regular
> expressions (except the nested comments of course).

before, the lexical structure was context free because of nested 
comments and floats of the form "[0-9]+\.". the latter can be matched 
with regexps if they support lookaheads, though.
if you stick to the specs verbatim, q"EOS...EOS" as a whole is a string 
literal. assuming that all tokens/lexemes are atomic, a lexer can't 
"look inside" the string literal. from that point of view, the lexical 
structure it's still context free.
if possible, i'd add a thin wrapper around an automatically generated 
lexer that checks the delimiters in a postprocess.