So... let's document dmd

[Walter Bright via Digitalmars-d]

On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:
> I disagree. I think having the dmd itself (lexer, parser, etc.) as a
> library (with the dmd executable merely being the default frontend) will
> do D a lot of good.
>
> For one thing, IDE's will no longer need to reinvent a D parser for the
> purposes of syntax highlighting;

On the other hand, using lexer.d and parse.d as a guide to build your own is a 
trivial undertaking. The Boost license is designed so this can be done without 
worrying about making a derived work.

I looked into doing syntax highlighting for my editor, MicroEmacs. It turns out 
it is not so easy to just use a compiler lexer/parser for it. For one thing, the 
one used in the compiler is optimized for speed in a forward pass through the text.

But a syntax highlighter in a text editor is different. Suppose I change a 
character in the middle of a line. All the highlighting from that point forward 
may change. And to figure out what that change is, the parser/lexer has to start 
over from the beginning of the file! (Think string literals, nested comments, 
quoted string literals, etc.) This would make editing slow.

I surmised that a solution is to have each line in the editor be tagged with a 
state to show the lexing state at the beginning of that line. Then, when a 
character in a line changes, the lexer can be restarted from that state, and it 
continues forward until the next line state matches the new computed state. This 
would make it enormously faster.

But the compiler's lexer is not designed to be restartable in the middle.

Similarly, a source code formatter would be different in a different way. It 
would need, for example, extra information about comments, token start/end 
positions, etc. Such data collection would be irrelevant to a compiler, and 
would slow it down and consume unneeded memory.