CTFE Overhaul (Was: RFC: Thrift project proposal (draft))

[Don]

dsimcha wrote:
> On 3/26/2011 4:16 PM, Don wrote:
>> dsimcha wrote:
>>> On 3/26/2011 12:16 PM, Don wrote:
>>>> I'm giving CTFE a *major* overhaul right now. I don't know if I'll be
>>>> finished in time for the next compiler release, but definitely by the
>>>> release after that. Most importantly, bug 1330, which is the root cause
>>>> of almost all of the problems, will be fixed. I hope to move CTFE out
>>>> the "experimental feature" category.
>>>
>>> This is great news, I'm looking forward to it. Thanks for the hard
>>> work. Out of curiosity, can you give a brief overview of what new
>>> things CTFE will be usable for?
>>
>> The basic problem with the current implementation of CTFE is that it
>> uses copy-on-write. This means that references (including dynamic
>> arrays) don't work properly -- they just copy a snapshot of the thing
>> they are referencing. This is bug 1330. It also means it burns up memory
>> like you wouldn't believe.
> 
> Right.  IIUC there's also no way to free the memory from copies that are 
> no longer referenced.  I can see where this would leak memory like a sieve.

That's not the big problem, actually. The issue is that x[7]=6;
duplicates x, even if x has 10K elements.
Now consider:
for(int i=0; i<x.length; ++i) x[i]=3;
// creates 100M new elements!! Should create none, or 10K at most.


>> I'm changing CTFE to use in-place modification. This fixes all those
>> issues. But this is obviously a fairly intense change, and will take
>> quite a lot of time to iron out all the corner cases. So that's all I'm
>> planning on doing right now.
> 
> This is a _huge_ improvement, but does it address the issue of freeing 
> memory or is that beyond the scope?

Outside the scope, but it will use an order of magnitude less memory in 
the first place, in the cases which are causing the biggest problems 
(such as the one I showed above).


>> But once that's done, it will be straightforward to implement other
>> reference types, such as classes and pointers (pointer arithmetic will
>> be restricted to pointers which point to array members). Once classes
>> are implemented, it's straightforward to do exceptions. So, pretty much
>> everything.
> 
> Excellent.
> 
>>
>> I've been planning on doing this for over a year, but while Walter was
>> working on 64-bit, I felt that I was the only one working on the
>> showstopper wrong-code bugs and regressions, so I put this
>> important-but-not-urgent stuff aside.
> 
> Agreed.  I love the 64-bit support (I've been using it for real work and 
> it's surprisingly solid) but the pace of fixing miscellaneous bugs was 
> understandably glacial while it was being implemented.