D port of dmd: Lexer, Parser, AND CodeGenerator fully operational

[Zach the Mystic]

On Thursday, 8 March 2012 at 02:32:11 UTC, Ary Manzana wrote:
>
> Very nice!
>
> I wonder, how did you port it?

Manually!
It was hard, but I'm new enough to programming in general that it 
was just as much of a learning process as a tedious grind or 
something.

> And also, how are you going to maintain it synchronized to the 
> lasted dmd version?

I'm under the impression that the syntax of the language changes 
much more rarely than the semantic and backend. While I don't 
have much experience with software maintenance, it will 
definitely be easier to maintain a mere parser than an entire 
semantic and backend.

I have two more things to say here, actually. One, you know, in a 
perfect world there would not even _be_ a D compiler written in 
C++ (i.e. only one written in D!), so I don't see why additions 
to this compiler have to be precisely like they are in dmd. They 
just need to be consistent with the language specification.

But two, I have a prospective use for this program which I am 
going to explore further. The idea is that the closer any given 
part of a compiler is to the human-readable code structure - i.e. 
the lexer is very close, the parser is a little further away, 
semantic even further, etc. - the more useful it will be as a 
development tool. As long as you can reverse any given 
compilation process - e.g. a parser is reversed by a code 
generator, a lexer by a "token generator", etc. - you can go 
backward and show the current state of the program to the user on 
the screen.

So instead of focusing on getting ddmd to work as a full-fledged 
compiler, I ( we? ) could focus on adding features which are 
useful for the development process. For example, say we are able 
to implement templates, which happen pretty soon after parsing, I 
believe, and also a reverser which allows us to put their current 
state to screen. Now we could look at the structure which WOULD 
be generated when compiled. CTFE may also be within reach, but 
I'm not at all sure because I know very little about it.

My immediate goal is to make an editor which doesn't work so much 
with text files as with fully parsed syntax trees, and merely 
converts them to code for displaying on the screen, and for 
writing to file to be sent to the actual D compiler.

Continuing with this theme, I've thought about some of the 
problems I might encounter ahead. But I'm rather excited about 
the solutions too. One obvious problem is that a parsed program 
is likely to be much more memory intensive than a text file. The 
solution I imagine to this problem is to literally convert chunks 
of a program BACK into text when they go offscreen. In other 
words, you're storing it as text, but editing it as a program 
tree. Not only would this text take up less memory, but it would 
also be easy to search through as text if that were the preferred 
editing style for the moment. It would also make writing the 
whole thing to file much faster, which will be extremely 
necessary because the whole process hinges on being able to 
quickly convert your tree to file so that it can be picked up by 
dmd, for example. Also, you could easily use any text editing 
capabilities you'd already implemented on non-D files.

Another thing I enjoy fantasizing about in this regard, is a 
program which duplicates its already parsed objects, hands them 
directly to its built-in compiler, and waits for the result 
without even needing to write to a file.

Anyway, the first thing I need is a gui, and a code generator 
capable of coloring its output appropriately, so I'm working on 
that, but it's not (even close to) ready for show yet!