[dmd-internals] dmd functions re-entrancy and other stuff

[Denis]

Hi, guys!

I'm currently integrating ddmd into an ide prototype I wrote a while
ago. Up until now it relied on external compilers (dmc and dmd) to
compile code and my current goal is to re-use ddmd for AST building
(for autocompletion) and (more importantly) code generation. Ideally,
I'd store every module in a parsed and fully semantically analyzed
form, and only call genobjfile() when I need to generate binary.

And I'm having a few issues that I'd like to share with you in hope
for an advice or two.

First of all, dmd heavily relies on global and static variables, but
that's easy to fix (I made most of the global __gshared for now, and
slowly moving them into a Global class later to be renamed into
something like CompileContext).
Next, some dmd methods are doing nothing when run for second time.
While this is exactly what you need for semantic/semantic2/semantic3,
it's not the case for toObjFile. For example, here is the code from
FuncDeclaration.toObjFile():

    if (semanticRun >= 5)	// if toObjFile() already run
        return;

I commented it out (for now), and it worked for me for a simple test case.

So here is my question #1: how correct that is? Do you really ever
need that check?
As an alternative, I could traverse AST and set semanticRun to 4 prior
to code generation, but that's something I'd like to avoid.
What solution is better?

Next, toObjFile() doesn't look to be re-entrable to me in general. For
example, I saw something like this:

assert(!vthis.csym);
vthis.toSymbol(); // result in creating vthis.csym

As a result, I can't run that code twice (it will fail for second time)

How correct that assertion is? Is it safe to remove it?

Could you please point me to other non-reentrable methods, and give
any hints on overcoming issues like this?

Anothing thing I'm thinking about is incremental compilation. I've
heard about many bugs in dmd that prevented it from working properly
(e.g. where should template body be emitted?) that resulted in a
xfBuild tool (which is build tool by h3r3tic based on dmd). I hope
I'll be able to fix these issues, too. What bothers me, however, is
dependency tracking. If some module 'A' has changed, everything that
uses it needs to be invalidated and recompiled (in general). However,
the change might be isolated to a function/class body (which is often
the case in every-day development) and as such a much smaller modules
subset really needs to be semantically re-analyzed and recompiled.

Other stuff I think about is parallel module compilation.
Theoretically, each dsymbol can be semantically analyzed independently
up to the point when it starts relying on other dsymbols
(synchronization point), and it doesn't looks like a hard feature to
implement.

These are long-term plans - there is a lot of work that has higher
priorities. What do I ask is *your* thoughts on it. Hopefully, some of
you already thought about it, and have something to share. Please do!
:)

Thanks in advance!