[dmd-internals] Fixing forward ref bugs for good

[Don Clugston]

On 15 September 2011 08:29, Rainer Schuetze <r.sagitario at gmx.de> wrote:
> On 15.09.2011 05:58, Benjamin Shropshire wrote:
>>
>> On 09/14/2011 12:06 AM, Rainer Schuetze wrote:
>>>
>>> On 14.09.2011 00:04, Walter Bright wrote:
>>>>
>>>> Don, just so you know, I've been thinking for a while about
>>>> transitioning from doing the semantic pass in order to doing it completely
>>>> "on demand". In other words, try to semantic a declaration. In the process,
>>>> any declarations it depends on are semantic'd if not already, recursively.
>>>
>>> I've been trying something similar for Visual D in its yet to integrate
>>> semantic analysis for intellisense. Still, static if and mixins get in the
>>> way of complete "on demand" handling. When a symbol is searched in a scope
>>> (e.g. a module, class, struct), some preparational work has to be done
>>> before the member list can be searched:
>>>
>>> 1. all "simple" non-scoping members are expanded (version/debug
>>> conditions, attributed declaration blocks). the branch inserted to the
>>> scopes' member list is also searched for "simple" non-scoping members (this
>>> step could also be done non-lazily, but doing it lazily slightly changes the
>>> interaction of version statements and conditionals with "static if"
>>> conditionals - good or bad, I don't know)
>>>
>>> 2. "complex" non-scoping members are expanded in lexical order (static
>>> if, mixins). When inserting the expanded branch into the scopes member list,
>>> the expansion restarts at 1.
>>>
>>> This works out better than the current dmd implementation, e.g. when
>>> forward referencing symbols in a mixin. There are still situations that
>>> depend on interpretation order, but that is to be expected when "static if"
>>> is used.
>>>
>>
>> Every time I've puzzled over the problem, the solution I've gravitated to
>> is to have the symbol table logic result be tri-state: symbol-found,
>> no-symbol, unknown/incomplete (for when a lookup includes an unprocessed
>> scope). From there, you greedily evaluate all symbol that you can and
>> proceed with whatever processing can be done, bailing when an "incomplete"
>> results is found and keeping a list of where to come back and try again
>> later. The only question then is how to handle the case where you dead lock.
>> I suspect that if you make that illegal, a lot of legacy code will break.
>> I'm going to guess we will want to have a small set of well thought out
>> deadlock escape rules.
>
> I guess, Walter also wants to get rid of the "try again later" part.
> Especially is-expressions and trait(compiles) are getting rather
> indeterministic and might depend on other symbols being looked up before. If
> the evaluation is lazy, there is some hope that dependencies will be cyclic
> only in cases that are actual errors. I'm not sure about a good
> cycle-detection, though. (Is that what you meant by "deadlock escape"?)

trySemantic is the big problem. The traits(compiles) thing is only
complicated because gagging is also used for trySemantic.

When traits(compiles) hits a forward reference, the correct logic is:
* if it is part of a speculative template instantiation, run semantic
on it with errors gagged;
* else, run normal semantic on it, without errors gagged.

But trySemantic is a big mess. To work properly it would need to be
able to undo _everything_, and I don't think that's feasible -- I
don't think we'd ever eliminate all the obscure bugs.

OTOH detecting cycles reliably in all cases might be nearly as difficult!

> What happens, if the evaluation of "static if" turns out to require symbols
> from the same scope? (Something I did not mention above: unconditionally
> existing or expanded members of a scope should be added to the symbol lookup
> as soon as possible.) My current suggestion is: do not recurse into the
> expansion of "complex" members, just use the currently available symbols.

What do you mean by "complex" members?