dmd Lexer and Parser in D

[Zach Tollen]

On 2/4/12 6:59 AM, F i L wrote:
> Very cool. I was talking with someone on the IRC about the
> possibility/difficulties of making DMD's parser/lexer/AST stay open in
> memory with protocols designed for IDE code-completion communication. It
> would be ideal to have an IDE's intellisense automatically update with
> DMD semantically.

This is my thinking too. One good thing about having cut the program is 
that it's a much lighter weight now, and I did it because I thought, 
well, maybe once it's paired down, I can actually steer it toward IDE 
functionality. For example, you could really cut out a lot of the 
members of the data structures which only point to backend functionality 
anyway.

Even if the whole project fails I won't regret doing it because I 
learned a lot about D in the process.

What I'm really wondering is if you wanted a program which helped you 
edit the syntax tree directly and only produced a text file for saving 
and running, what kind of data structure would you like to have 
representing the syntax tree? Without knowing anything else, I guessed 
that it would be nice to have something resembling the official D 
parse-tree.

> Unfortunately the conclusion was that it would be to difficult an
> undertaking to be realistic, since DMD is designed to be run-and-done
> (also something about "Walter code" :-)).

I was wondering if you couldn't take a parse-tree data structure and 
deparse (disparse?) it back to formatted program code so that you could 
see what you were editing? As unrealistic as that sounds, I'm 
sufficiently attracted to the idea that I'm investigating it with an 
open mind.

 > But maybe a rewrite/port of DMD, especially one written in D, might
 > be able to be reworked with this
 > goal in mind? How complete is DDMD?

This is exactly what I'm aiming at. My basic hopes for its being 
possible are the comforting notion that the huge part of dmd is actually 
the stuff I threw out! The goal would be to construct the front end (of 
a front end) which was at least theoretically capable both of allowing 
code editing, and of translation to a more backend-friendly data 
structure. If that's not possible, then I'm stuck with this thought that 
you edit the tree, then the IDE reverses the parse back into ordinary 
code for saving and compiling.

If anybody can refer me to any examples and demonstrations of this type 
of code-editing, please do. As someone new to programming I'm really 
wondering why, if the program itself is understood by the computer as a 
tree, why do I have to edit a text file instead of a tree?

Zach