DIP 1025--Dynamic Arrays Only Shrink, Never Grow--Community Review Round 1

Jonathan M Davis newsgroup.d at jmdavisprog.com
Mon Nov 11 22:40:44 UTC 2019


On Monday, November 11, 2019 2:41:14 PM MST Walter Bright via Digitalmars-d 
wrote:
> Memory safety cannot be achieved without control over who is the owner of
> memory.

And how does this DIP help with that? Dynamic arrays do not know or care
where their memory came from, and they do not track lifetimes in any way,
shape, or form. Doing anything that might append to them then guarantees
that they're allocated by the GC, because that's the only way that they can
grow. Without doing one of those operations, you would have to actually look
at the ptr value of the dynamic array and ask the GC whether it's owned by
the GC. Getting rid of the append operations does not change that. It's
still impossible to know who owns the memory by simply looking at the type,
and code that passes slices of non-GC allocated memory around has to be
aware of the lifetime of those dynamic arrays and what's done with them to
know whether any dynamic arrays referring to that memory still exist. The
type system doesn't help with that beyond stuff like scope and pure limiting
what can be done with the dynamic array that gets passed in. And getting rid
of the append operations doesn't change that.

Sure, right now, if you pass a dynamic array which is a slice of malloc-ed
memory to a function, that function could append to that dynamic array and
result in a reallocation. Why is that a problem? It's not like that's going
to cause a memory leak unless it was your only pointer to that memory, and
if that's the case, having the dynamic array shrink on you would be just as
big a problem, if not bigger. Either way, you have to know what the code is
doing with the dynamic array if you're going to try to determine when it's
safe to free that memory, because dynamic arrays can normally be passed
around and copied with impunity, and they're not reference-counted. Their
very nature pretty requires that if you're going to have a dynamic array
which slices non-GC-allocated memory, you have to carefully control what you
do with it, because nothing about the dynamic array manages the lifetime of
the memory it points to. That's the case whether appending is allowed or
not.

If for some reason, it is actually a problem for a piece of code if a
dynamic array ever gets appended to, then requiring that the code be @nogc
fixes that. It's then not possible to pass the dynamic array to any
functions which might append to it.

Really, code that's using dynamic arrays which are slices of anything other
than GC-allocated memory has to be very limited in how it passes them around
regardless of whether appending is possible, because dynamic arrays don't
manage their own memory at all. Code that wants to do much passing around of
malloc-ed memory to anything other than a pure function where it's clear
that the slices can't be squirreled away in any of the arguments or return
value is going to tend to need to wrap that memory in something other than a
dynamic array in order to manage its lifetime - either that, or it needs to
be written in a way that the programmer can clearly know that anything that
even might contain a reference to that memory is long gone before they're
looking to free that memory (e.g. because it's allocated at the beginning of
the program and freed at the end).

It looks to me like you're crippling the normal use case in attempt to deal
with a less common one. And it doesn't look to me like it's really fixing
anything for that less common use case.

Honestly, if it really is the case that we have to lose the ability to
append to dynamic arrays in order to get @safety for malloc-ed memory, then
I'd rather give up on @safety for malloc-ed memory and keep the ability to
append to dynamic arrays. But from what I can tell, the only argument that
the DIP has for why appending to dynamic arrays even could be a problem is
because it might be problematic if a dynamic array which slices
non-GC-allocated memory ends up being copied into newly allocated GC memory,
because it's appended to. I dispute that that's a real problem, and even if
it is one, @nogc prevents it without any language changes.

It should be clear to anyone seriously using dynamic arrays that

int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice ~= 1;
free(slice.ptr); // Oops!

is a problem. It's a given that appending can cause a reallocation. But even
if appending weren't allowed, you could still have something like

int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice = slice[1 .. $];
free(slice.ptr); // Oops!

and have problems, because the dynamic array isn't slicing the beginning of
that block of memory anymore. So, allowing a dynamic array to shrink (or
even be assigned to at all) causes similar problems as appending in this
case. Either way, relying on the dynamic array itself to know which pointer
to use to free that memory is just plain bad practice unless the code is
definitely never going to do anything that would mutate the dynamic array
itself (be it slicing, appending, or assignment from another array). In
general, if code is going to slice malloc-ed memory to get a dynamic array,
it should be keeping around a separate pointer which refers to the block
that was allocated so that it can free it when no more dynamic arrays exist
which point to that memory rather than relying on the dynamic array for that
information. Either way, the general inability to track the lifetime of
dynamic arrays outside of things like scope and pure means that you can't
rely on the type system to track the lifetime of malloc-ed memory outside of
functions that simply take an argument and return a result (with scope and
pure used to ensure that no slices of that memory are holed away somewhere).

As for code such as

enum { dead, alive }
int[] cat = new int[6];
cat[5] = alive;
int[] b = cat;
b ~= 1;      // may or may not move b to new location
b[5] = dead; // indeterminate whether cat[5] is dead or alive

So what? It's a given that appending to a dynamic array could cause a
reallocation. If that's a problem, the code can check the array's capacity
first to see if appending would cause a reallocation. This behavior of
arrays is nothing new and isn't specific to dynamic arrays which are slices
of memory which wasn't allocated by the GC. If the code actually cares about
whether a reallocation takes place, it can easily check whether it will or
has taken place, and it can easily prevent it from taking place by requiring
that the code be @nogc. The DIP does nothing to show why this behavior is
actually a problem for @safe code.

- Jonathan M Davis





More information about the Digitalmars-d mailing list