DIP 1025--Dynamic Arrays Only Shrink, Never Grow--Community Review Round 1

[Exil]

On Monday, 11 November 2019 at 22:40:44 UTC, Jonathan M Davis 
wrote:
> On Monday, November 11, 2019 2:41:14 PM MST Walter Bright via 
> Digitalmars-d wrote:
>> Memory safety cannot be achieved without control over who is 
>> the owner of memory.
>
> And how does this DIP help with that? Dynamic arrays do not 
> know or care where their memory came from, and they do not 
> track lifetimes in any way, shape, or form. Doing anything that 
> might append to them then guarantees that they're allocated by 
> the GC, because that's the only way that they can grow. Without 
> doing one of those operations, you would have to actually look 
> at the ptr value of the dynamic array and ask the GC whether 
> it's owned by the GC. Getting rid of the append operations does 
> not change that. It's still impossible to know who owns the 
> memory by simply looking at the type, and code that passes 
> slices of non-GC allocated memory around has to be aware of the 
> lifetime of those dynamic arrays and what's done with them to 
> know whether any dynamic arrays referring to that memory still 
> exist. The type system doesn't help with that beyond stuff like 
> scope and pure limiting what can be done with the dynamic array 
> that gets passed in. And getting rid of the append operations 
> doesn't change that.
>
> Sure, right now, if you pass a dynamic array which is a slice 
> of malloc-ed memory to a function, that function could append 
> to that dynamic array and result in a reallocation. Why is that 
> a problem? It's not like that's going to cause a memory leak 
> unless it was your only pointer to that memory, and if that's 
> the case, having the dynamic array shrink on you would be just 
> as big a problem, if not bigger. Either way, you have to know 
> what the code is doing with the dynamic array if you're going 
> to try to determine when it's safe to free that memory, because 
> dynamic arrays can normally be passed around and copied with 
> impunity, and they're not reference-counted. Their very nature 
> pretty requires that if you're going to have a dynamic array 
> which slices non-GC-allocated memory, you have to carefully 
> control what you do with it, because nothing about the dynamic 
> array manages the lifetime of the memory it points to. That's 
> the case whether appending is allowed or not.
>
> If for some reason, it is actually a problem for a piece of 
> code if a dynamic array ever gets appended to, then requiring 
> that the code be @nogc fixes that. It's then not possible to 
> pass the dynamic array to any functions which might append to 
> it.
>
> Really, code that's using dynamic arrays which are slices of 
> anything other than GC-allocated memory has to be very limited 
> in how it passes them around regardless of whether appending is 
> possible, because dynamic arrays don't manage their own memory 
> at all. Code that wants to do much passing around of malloc-ed 
> memory to anything other than a pure function where it's clear 
> that the slices can't be squirreled away in any of the 
> arguments or return value is going to tend to need to wrap that 
> memory in something other than a dynamic array in order to 
> manage its lifetime - either that, or it needs to be written in 
> a way that the programmer can clearly know that anything that 
> even might contain a reference to that memory is long gone 
> before they're looking to free that memory (e.g. because it's 
> allocated at the beginning of the program and freed at the end).
>
> It looks to me like you're crippling the normal use case in 
> attempt to deal with a less common one. And it doesn't look to 
> me like it's really fixing anything for that less common use 
> case.
>
> Honestly, if it really is the case that we have to lose the 
> ability to append to dynamic arrays in order to get @safety for 
> malloc-ed memory, then I'd rather give up on @safety for 
> malloc-ed memory and keep the ability to append to dynamic 
> arrays. But from what I can tell, the only argument that the 
> DIP has for why appending to dynamic arrays even could be a 
> problem is because it might be problematic if a dynamic array 
> which slices non-GC-allocated memory ends up being copied into 
> newly allocated GC memory, because it's appended to. I dispute 
> that that's a real problem, and even if it is one, @nogc 
> prevents it without any language changes.
>
> It should be clear to anyone seriously using dynamic arrays that
>
> int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
> slice ~= 1;
> free(slice.ptr); // Oops!
>
> is a problem. It's a given that appending can cause a 
> reallocation. But even if appending weren't allowed, you could 
> still have something like
>
> int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
> slice = slice[1 .. $];
> free(slice.ptr); // Oops!
>
> and have problems, because the dynamic array isn't slicing the 
> beginning of that block of memory anymore. So, allowing a 
> dynamic array to shrink (or even be assigned to at all) causes 
> similar problems as appending in this case. Either way, relying 
> on the dynamic array itself to know which pointer to use to 
> free that memory is just plain bad practice unless the code is 
> definitely never going to do anything that would mutate the 
> dynamic array itself (be it slicing, appending, or assignment 
> from another array). In general, if code is going to slice 
> malloc-ed memory to get a dynamic array, it should be keeping 
> around a separate pointer which refers to the block that was 
> allocated so that it can free it when no more dynamic arrays 
> exist which point to that memory rather than relying on the 
> dynamic array for that information. Either way, the general 
> inability to track the lifetime of dynamic arrays outside of 
> things like scope and pure means that you can't rely on the 
> type system to track the lifetime of malloc-ed memory outside 
> of functions that simply take an argument and return a result 
> (with scope and pure used to ensure that no slices of that 
> memory are holed away somewhere).
>
> As for code such as
>
> enum { dead, alive }
> int[] cat = new int[6];
> cat[5] = alive;
> int[] b = cat;
> b ~= 1;      // may or may not move b to new location
> b[5] = dead; // indeterminate whether cat[5] is dead or alive
>
> So what? It's a given that appending to a dynamic array could 
> cause a reallocation. If that's a problem, the code can check 
> the array's capacity first to see if appending would cause a 
> reallocation. This behavior of arrays is nothing new and isn't 
> specific to dynamic arrays which are slices of memory which 
> wasn't allocated by the GC. If the code actually cares about 
> whether a reallocation takes place, it can easily check whether 
> it will or has taken place, and it can easily prevent it from 
> taking place by requiring that the code be @nogc. The DIP does 
> nothing to show why this behavior is actually a problem for 
> @safe code.
>
> - Jonathan M Davis

I mean there were warning signs in DIP 1021. But you know what, 
this DIP 1025 is starting to grow on me. I like it. The DIP is 
absolutely perfect, nothing needs to change about it! I trust 
Walter completely, after all "he has a pretty darn good one 
[track record]".