So... let's document dmd

[Timon Gehr via Digitalmars-d]

On 05.04.2016 10:46, Walter Bright wrote:
> On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:
>> I disagree. I think having the dmd itself (lexer, parser, etc.) as a
>> library (with the dmd executable merely being the default frontend) will
>> do D a lot of good.
>>
>> For one thing, IDE's will no longer need to reinvent a D parser for the
>> purposes of syntax highlighting;
>
> On the other hand, using lexer.d and parse.d as a guide to build your
> own is a trivial undertaking. The Boost license is designed so this can
> be done without worrying about making a derived work.
>
> I looked into doing syntax highlighting for my editor, MicroEmacs. It
> turns out it is not so easy to just use a compiler lexer/parser for it.
> For one thing, the one used in the compiler is optimized for speed in a
> forward pass through the text.
>
> But a syntax highlighter in a text editor is different. Suppose I change
> a character in the middle of a line. All the highlighting from that
> point forward may change. And to figure out what that change is, the
> parser/lexer has to start over from the beginning of the file! (Think
> string literals, nested comments, quoted string literals, etc.) This
> would make editing slow.
> ...

Also, tools might want to parse a more intuitive grammar in order to be 
able to give suggestions on how to adapt the code such that DMDs parser 
will accept it.

> I surmised that a solution is to have each line in the editor be tagged
> with a state to show the lexing state at the beginning of that line.
> Then, when a character in a line changes, the lexer can be restarted
> from that state, and it continues forward until the next line state
> matches the new computed state. This would make it enormously faster.
> ...

Some additional care will need to be taken, e.g.

/+ <backspace> <backspace>

or even simply

/+ comment +/

> But the compiler's lexer is not designed to be restartable in the middle.
>
> Similarly, a source code formatter would be different in a different
> way. It would need, for example, extra information about comments, token
> start/end positions, etc. Such data collection would be irrelevant to a
> compiler, and would slow it down and consume unneeded memory.
>