Feature suggestion: in-place append to array

[grauzone]

Steven Schveighoffer wrote:
> On Thu, 04 Mar 2010 12:43:32 -0500, grauzone <none at example.net> wrote:
> 
>> Steven Schveighoffer wrote:
>>> On Thu, 04 Mar 2010 11:43:27 -0500, grauzone <none at example.net> wrote:
>>>
>>>> Some sort of "resetAndReuse" function to clear an array, but 
>>>> enabling to reuse the old memory would be nice:
>>>>
>>>> int[] a = data;
>>>> a = null;
>>>> a ~= 1; //reallocates (of course)
>>>> a.length = 0;
>>>> a ~= 1; //will reallocate (for safety), used not to reallocate
>>>> resetAndReuse(a);
>>>> assert(a.length == 0);
>>>> a ~= 1; //doesn't reallocate
>>>>
>>>> This can be implemented by setting both the slice and the internal 
>>>> runtime length fields to  0.
>>>>
>>>> Additionally, another function is necessary to replace the old 
>>>> preallocation trick:
>>>>
>>>> //preallocate 1000 elements, but don't change actual slice length
>>>> auto len = a.length;
>>>> a.length = len + 1000;
>>>> a.length = len;
>>>>
>>>> As I understood it, this won't work anymore after the change. This 
>>>> can be implemented by enlarging the array's memory block without 
>>>> touching any length fields.
>>>>
>>>> I'm sure the function you had in mind does one of those things or both.
>>>  proposed usage (as checked in a couple days ago):
>>>  int[] a;
>>> a.setCapacity(10000); // pre-allocate at least 10000 elements.
>>> foreach(i; 0..10000)
>>>    a ~= i; // no reallocation
>>> a.length = 100;
>>> a.shrinkToFit(); // resize "allocated" length to 100 elements
>>> a ~= 5; // no reallocation.
>>
>> What shrinkToFit() does is not really clear. Does it reallocate the 
>> memory block of the array such, that no space is wasted? Or does it 
>> provide (almost) the same functionality as my resetAndReuse(), and 
>> make the superfluous trailing memory available for appending without 
>> reallocation?
> 
> Sorry, should have added:
> 
> assert(a.length == 101);
> 
> Basically, shrinkToFit shrinks the "allocated" space to the length of 
> the array. To put it another way, you could write your resetAndReuse 
> function as follows:
> 
> void resetAndReuse(T)(ref T[] arr)
> {
>   arr.length = 0;
>   arr.shrinkToFit();
> }
> 
> I want to avoid assuming that shrinking the length to 0 is the only 
> usable idiom.

Ah, great.

>>
>> I think a resetAndReuse is really needed. I have found it can prevent 
>> "GC thrashing" in many cases. E.g. when caching frequently 
>> re-evaluated data in form of arrays (free previous array, then 
>> allocate array that isn't larger than the previous array).
> 
> Yes, it is useful in such cases.  The only questionable part about it is 
> that it allows for stomping in cases like:
> 
> auto str = "hello".idup;
> auto str2 = str[3..4];
> str2.shrinkToFit();
> str2 ~= "a";
> assert(str == "hella");
> 
> So it probably should be marked as unsafe.

Doesn't it conform to Andrei's ideas about memory safety? It can stomp 
over other data, but it can't be used to subvert the type system. Also, 
it's not the default behavior: if you don't use this function, stomping 
can never happen.

But it must be disabled for arrays of immutable types (i.e. strings).

> Again, the name shrinkToFit isn't my favorite, ideas welcome.

stompOnAppend()? uniqueSlice()? trashRemainder()?

I don't know either, all sound a bit silly.

PS: I'm glad to hear that your patch is supposed to be included in the 
next release. Still waiting for dsimcha's one.

> -Steve