DMD 1.021 and 2.004 releases

[Kirk McDonald]

Jari-Matti Mäkelä wrote:
> Kirk McDonald wrote:
> 
> 
>>Walter Bright wrote:
>>
>>>Stewart Gordon wrote:
>>>
>>>
>>>>Maybe.  But still, nested comments are probably likely to be supported
>>>>by more code editors than such an unusual feature as delimited strings.
>>>
>>>
>>>Delimited strings are standard practice in Perl. C++0x is getting
>>>delimited strings. Code editors that can't handle them are going to
>>>become rapidly obsolete.
>>>
>>>The more unusual feature is the token delimited strings.
>>
>>Which, since there's no nesting going on, are actually very easy to
>>match. The Pygments lexer matches them with the following regex:
>>
>>q"([a-zA-Z_]\w*)\n.*?\n\1"
> 
> 
> It's great to see Pygments handles so many possible syntaxes. Unfortunately
> backreferences are not part of regular expressions. I've noticed two kinds
> of problems in tools:
> 
> a) some can't handle backreferences, but provide support for nested comments
> as a special case. So comments are no problem then, but all delimited
> strings are.
> 
> b) some lexers handles both nested comments and delimited strings, but all
> delimiters must be enumerated in the language definition. Even worse, some
> highlighters only handle delimited comments, not strings.
> 
> Maybe the new features (= one saves on average < 5 characters of typing per
> string) are more important than tool support? Maybe all tools should be
> rewritten in Python & Pygments?

While D now requires a fairly powerful lexer to lex properly, it's still 
easier to lex than, for example, Ruby. Ruby's heredoc strings are more 
complicated than D's. Even Pygments requires some advanced callback 
trickery to lex them properly.

Docs on Ruby's "here document" string literals:
http://docs.huihoo.com/ruby/ruby-man-1.4/syntax.html#here_doc

Pygments's Ruby lexer:
http://trac.pocoo.org/browser/pygments/trunk/pygments/lexers/agile.py#L260

Also, the lexical phase is still entirely independent of the syntactical 
and semantic phases, even if it is a little more difficult than it was 
before.

My point is simply that any tool capable of lexing Ruby -- and there are 
a number of these -- is more than powerful enough to lex D. So the bar 
is high, but quite reachable.

I do not think it is extraordinary that a tool written in Python would 
take advantage of Python's regular expressions' features.

-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org