Mutable enums

[Timon Gehr]

On 11/15/2011 04:53 PM, Steven Schveighoffer wrote:
> On Mon, 14 Nov 2011 16:28:52 -0500, Timon Gehr <timon.gehr at gmx.ch> wrote:
>
>> On 11/14/2011 09:39 PM, Steven Schveighoffer wrote:
>>> Look at the code generated for enum a = [1, 2, 3]. using a is replaced
>>> with a call to _d_arrayliteral. There is no CTFE going on.
>>>
>>
>> There is some ctfe going on, but the compiler has to allocate the
>> result anew every time it is used. So there is also some runtime
>> overhead.
>>
>> To make my point clearer:
>>
>> int foo(){return 100;}
>> enum a = [foo(), foo(), foo()]; // a is the array literal [100, 100,
>> 100];
>>
>> void main(){
>> auto x = a; // this does *not* call foo. But it allocates a new array
>> literal
>> }
>
> Yes, you are right. The issue is that the resulting array is initialized
> at runtime, not that CTFE is being avoided. After doing some of these
> tests, I have a better understanding of the issues.
>
>>> The compiler has no choice. It must develop the array at runtime, or
>>> else the type allows one to modify the source value (just like in D1 how
>>> you could modify string literals). In essence, the compiler is creating
>>> a new copy for every usage (and building it from scratch).
>>>
>>
>> That is a quality of implementation issue. The language semantics do
>> not require that.
>
> The language semantics require that if an enum type points at mutable
> data, a runtime allocation *must* occur to avoid corruption of literals.
> I think a rule requiring an enum to be immutable or implicitly cast to
> immutable puts the burden of runtime allocation on the coder, making it
> clear what's going on.
>
> In C++, novice coders typically pass classes by value not knowing what a
> horrible thing this is doing. Then they are puzzled why the code is so
> slow, the syntax is so short! This is another case of a hidden
> allocation which can be either avoided or made visible.
>
>>>> enum a = [2,1,4];
>>>> enum b = sort(a); // should be fine.
>>>
>>> I was actually surprised that this compiles. But this should not be a
>>> problem even if a was immutable(int)[]. sort should be able to create a
>>> copy of an immutable array in order to sort it. It doesn't matter the
>>> performance hit, because this should all be done at compile time.
>>>
>>
>> It does not, but explicitly calling .dup works
>> immutable x = [3,2,1];
>> immutable y = sort(x.dup);
>
> I'm saying sort (or another symbol, ctfesort?) can be made to do the dup
> automatically for you so you don't have to have it when using ctfe.
> Extra allocations during CTFE cost nothing (well, with a properly GC'd
> compiler, that is).
>
> Update: I have a better idea, see below.
>
>>> When I see an enum, I think "evaluated at compile time". No matter how
>>> complex it is to build that value, it should be built at compile-time
>>> and *used* at runtime. No complex function calls should be done at
>>> runtime, an enum is a value.
>>
>> Exactly. Therefore you assign from it by copying it.
>>
>> Compare to static array.
>>
>> int[10] x = [1,2,3,4,5,6,7,8,9,0];
>>
>> x still needs to be initialized at runtime.
>
> Yes, but this is spelled out because copying a static array requires
> moving data. However, this does *not* require a hidden allocation (even
> though it does do a hidden allocation currently).
>
> I'm not worried about copying data as much as I am about hidden
> allocations. Hidden allocations are a huge drag on performance. Every
> time you allocate, you need to take a global GC lock, and it's an
> unbounded operation (doing one allocation could run a collection cycle).

You don't actually _need_ a global GC lock. It is just how it is 
implemented in this case. Note that this is an explicit allocation:

int[] a = [1,2,3]; // just as explicit as a NewExpression

Only the enums "hide" it sometimes, but actually you should know the 
involved types.

>
>>> I did an interesting little test:
>
> [snip]
>
>>
>> That just tells us that DMD sucks at generating code for array literals.
>>
>> This generates identical code:
>>
>> import std.stdio;
>>
>> void main() {
>> writeln([1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3,
>> 3, 3, 3, 3]);
>> }
>>
>> You don't need enums for that.
>>
>>
>> What it actually should for both our examples is more like the following:
>>
>> import std.stdio;
>>
>> immutable _somewhereinrom = [1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2,
>> 2, 2, 3, 3, 3, 3, 3, 3, 3, 3];
>>
>> void main() {
>> writeln(_somewhereinrom.dup);
>> }
>>
>> push %ebp
>> mov %esp,%ebp
>> pushl 0x8097184
>> pushl 0x8097180
>> mov $0x80975c8,%eax
>> push %eax
>> call 8079470 <_adDupT>
>> add $0xc,%esp
>> push %edx
>> push %eax
>> call 807041c <_D3std5stdio15__T7writelnTAiZ7writelnFAiZv>
>> xor %eax,%eax
>> pop %ebp
>> ret
>>
>>
>> If writeln would actually be const correct, the compiler could even
>> get rid of the allocation.
>
> That is the idea. Get rid of the hidden allocation. Writeln *is* const
> correct, it can certainly print immutable(int)[].

Well, there is a function called writeln that can do that. That is a 
different function. But the one that gets actually called is not const 
correct as well.


This is writeln:

// Most general instance
void writeln(T...)(T args)
if (T.length > 1 || T.length == 1 && !is(typeof(args[0]) : const(char)[]))
{
     stdout.write(args, '\n');
}


=>
writeln([1,2,3]);
// modulo IFTY:
writeln!(int[])([1,2,3]);
// const correct?
writeln!(int[])([1,2,3].idup); // nope!

Error: cannot implicitly convert expression (_adDupT(& 
_D11TypeInfo_Ai6__initZ,[1,2,3])) of type immutable(int)[] to int[]


Now if there was a const there, the compiler could infer that writeln 
will not change the .duped array. Ergo it could pass the reference to 
immutable data directly. It would maybe help against template code bloat 
a bit, but not that much because const(immutable(int)[]) and the like 
are distinct types to const(int[]).




> The issue is not
> writeln, it's what the type of the array literal/enum is.
>

> Technically, an array literal is equivalent to an enum, and should
> follow the same rules.
>

Remember that immutable is transitive. That can really get in your way 
in this case.



>> This is not about enums that much, it is about array literals.
>>
>> The fact that stack static array initialization allocates is one of
>> DMDs bigger warts.
>>
>> Look at the ridiculous code generated for the following example:
>>
>> void main() {
>> int[24] x = [1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3,
>> 3, 3, 3, 3, 3];
>> writeln(x);
>> }
>
> Yes, these are all cases of the same issue.
>
>>>> enum a = [2,1,4];
>>>> enum b = sort(a.dup); // what exactly is that 'a.dup' thing?
>>>
>>> I don't think .dup should be necessary at compile time. Creating a
>>> sorted copy of an immutable array should be quite doable.
>>>
>>
>> I agree, phobos won't currently do it though.
>
> This is easily fixed. But maybe there is a better way (see below).
>
>>>> enum d = sort(c); // does not work?
>>>>
>>>> enum e = foo(a.dup, b.dup, c.dup, d.dup);
>>>
>>> Again, I don't think .dup would be used for dependent enums, I was
>>> rather thinking dup would be used where you need a mutable copy of an
>>> array during enum usage in normal code.
>>>
>>
>> But if the type of a,b,c,d is immutable(int)[] and foo is a function
>> that takes 4 int[]s then the .dup's are necessary to pass type checking.
>
> What about this idea:
>
> At a global level, expressions that result in CTFE being triggered, can
> be implicitly cast from mutable to immutable and vice versa via a
> deep-dup. This allows you to use enums as parameters to functions
> accepting mutable references. Then enums that are derived from other
> enums do not need to follow the same rules as runtime code that uses the
> enums.
>
> This of course, only happens at the global-expressions level, as
> function internals must compile at runtime as well.
>
> What I thought of as a solution was to create CTFE only functions that
> wrap other functions to do a dup. But you wouldn't want to do this
> during runtime, because dup is expensive. During compile time, dup costs
> nothing. This idea essentially takes the place of that boilerplate code.
>
> -Steve


That could work, but I think this is cluttering up the semantics a bit.