[dmd-internals] Fw: Fixing forward ref bugs for good

Don Clugston dclugston at googlemail.com
Fri Sep 23 00:08:07 PDT 2011


On 17 September 2011 08:00, Rainer Schuetze <r.sagitario at gmx.de> wrote:
> On 9/16/2011 1:03 PM, Don Clugston wrote:
>>>
>>> ----- Forwarded Message -----
>>> From: Steve Schveighoffer<schveiguy at yahoo.com>
>>> What I mean is, it feels like the wrong approach to ordering.  I'd rather
>>> ordering be dependent on ordering, not nesting.  It seems like a
>>> counter-intuitive way to do ordering that has the same issues using the
>>> order of declaration.
>>>
>>> People are going to look at some file that has static if(true) (or
>>> several
>>> levels of them!) and scratch their heads, potentially removing it.
>>> Are we trying to say here, ordering matters only for static ifs?
>>
>> The problem isn't static if, or even ordering, per se. It's the
>> combination of (a) reflection, and (b) the fact that declarations can
>> be conditionally added via static if and mixin; the two concepts are
>> fundamentally incompatible.
>> Any reflection that checks for existence of a symbol has a result
>> which can change with time.
>>
>> So, although the spec says that order of declarations doesn't matter, it
>> isn't:
>>
>> enum b1 = is(typeof(foo));
>> static if (!b2)  int foo;
>> enum b2 = is(typeof(foo));
>>
>> Is b1 true, or false? Currently, it's false, but becomes true if moved
>> to the bottom of the file.
>>
>> You could say that any use of a non-existent symbol potentially
>> depends on all non-evaluated static ifs; you don't know it doesn't
>> exist until you've expanded them ALL.
>> So one idea would be to evaluate every static if that had a condition
>> that didn't involve reflection. If you find a condition which is
>> blocked, you stop, and go onto the next one. Moderately difficult to
>> implement, but possible.
>> But often you get multiple static ifs which are blocked. They are
>> probably independent, but you don't know. What do you do? Well, you
>> could arbitrarily say you do them in the order the static ifs appear
>> in the file.
>> But it's not very predictable, because you can't just look at a static
>> if statement and see if it is blocked or not -- the reflection could
>> be happening anywhere in the file. I don't think that's viable.
>>
>>>  Then why
>>> can't the ordering just be the order they appear?  Is there any advantage
>>> to
>>> use nesting?
>>
>> The nesting behaviour is a natural consequence of saying that the
>> order doesn't matter. It's not a design decision. It does give an
>> advantage over order-in-file in the case of things like struct members
>> (where order in the file DOES matter) because you can control the
>> order the static ifs are evaluated (you can make the first static if
>> in the struct only be evaluated after all the others have finished).
>>
>>>   What about some sort of pragma or @annotation to define
>>> ordering?
>>
>> That's possible, too. (That's what I meant by syntax sugar).
>>
>>> And what about repeatedly trying evaluating a static if until it can be
>>> evaluated, or you are stuck?  Has that been considered and rejected for
>>> some
>>> reason?
>>
>> See above. It's not feasible.
>>
>>> Again, not understanding all the issues, so maybe it's just the only way
>>> possible.  But it sure looks bad.
>>
>> Fundamentally we're doing something which is contradictory (but so
>> useful in practice that we desperately want to keep it!) I think a
>> perfect solution is impossible.
>>
>> I think we have four feasible solutions:
>> (1)
>> static ifs and mixins are expanded in the order they appear in the
>> file. This is applied recursively until none are left.
>> Finally, everything else is evaluated in parallel.
>> (2)
>> static ifs and mixins are expanded in parallel. This is repeated until
>> none are left.
>> Finally, everything else is evaluated in parallel.
>> (3)
>>  Everything is evaluated in parallel, except for static ifs and mixins.
>>  If no static ifs or mixins, quit.
>>  static ifs and mixins are evaluated&  expanded in the order they
>> appear in the file. Repeat.
>> (4)
>>  Everything is evaluated in parallel, including static ifs and mixins
>> (but they aren't expanded).
>>  If no static ifs or mixins, quit.
>> static ifs and mixins are expanded. Repeat.
>>
>> All of these have surprising behaviour in some ways.
>> -Don.
>>
>
> I agree with Don's compilation. I just want to point out that the expansion
> does not need to recurse into nested scopes during the steps above (if there
> are no explicite symbol lookups during condition/mixin evaluation). All we
> want is the *complete* list of symbols in the current scope.
> Looking into nested scopes might get delayed by lazy evaluation, maybe even
> skipped in a fast compilation mode.

It looks as though we want a semantic pass 0, which evaluates all of
the mixins and static ifs.
Then the reflection issue is resolved as:
* any reflection which is evaluated while expanding a mixin or a
static if condition, represents an intermediate, potentially unstable
state, while symbols are still being generated.
* All other reflection represents the state after all symbols are added.

We can still have some pretty weird behaviour, where an expression is
used in a static if, but the same expression is different everywhere
else. Eg,
template Foo(int X)
{
     bool Foo = is(typeof(bar));
}
static if (!Foo!(2))  int bar;

static assert( Foo!(1) == Foo!(2) );  // fails!

We probably have to live with that.
The only alternative I can think of, is to dump the symbol table at
the end of the semantic0 pass, so that _all_ values are re-evaluated,
and so that you don't have to worry about whether something was used
in a static if, or not. This would be implemented, once again, by
storing the const-folding result into the mixins and static if's, and
then clearing the symbol table for that scope. This would mean that
static if and mixins are always 'snapshots' of the state of
compilation, but everything else represents the final state. With my
suggested 'do everything in parallel' you'd only have to clear the
symbol table once, but if they are evaluated in order, you need to
wipe the symbol table after every evaluation of a static if or mixin.
But the impact on compilation time could be kept small, by only wiping
the symbol table if you had a reflection result which might be
different to the final one; ie, if a is(typeof()) returned false. It's
quite a rare situation anyway.
I don't know if these semantics are sufficiently better to justify the
implementation difficulty.

But regardless, given we that we have to have a special semantic pass,
specifically for mixin declarations and static if, then the beauty of
have everything be order-independent is already damaged. So I think we
might as well say that the semantic0 "add symbols into scope" pass
applies only to static if and mixins, and is run in the order in which
the static ifs and mixins appear in the file; but the semantic meaning
of declarations themselves is independent of order.

So I would recommend option (1) from my list above: do static if and
mixin in order, then when they're finished, do the declarations in any
order.


More information about the dmd-internals mailing list