DIP 1025--Dynamic Arrays Only Shrink, Never Grow--Community Review Round 1

[Jonathan M Davis]

On Tuesday, November 12, 2019 1:43:27 AM MST Robert Schadek via Digitalmars-
d wrote:
> On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:
> > Although malloc() is marked as @system, it actually is @safe.
> > It's free() that needs to be @system.
>
> I think that is a mute point.
> malloc gives you a void*, you can't really do anything with that
> void*
> in @safe code.
> You can't even cast it to int*, and it shouldn't.
>
> I know I sound like a broken record, but IMO mmm and @safe is the
> wrong direction.

Honestly, I don't think that it can really be done without seriously
hamstringing ourselves.

Dynamic arrays and manually managed memory are a _terrible_ combination,
because dynamic arrays do nothing to manage lifetimes. They're really not
any different from passing naked pointers around as far as lifetime tracking
goes. This means that the only way that you can automatically manage that is
to restrict where they can be stored. scope does that, but the result is
that you can pretty much just pass the data into a function that operates on
it and doesn't store it. It doesn't even allow you to do something as simple
as have the function return a range wrapper like a lot of Phobos does. And
real world code typically needs to store stuff, not just pass it around.
Maybe what Walter is looking to do with @live will let it go a bit further,
but I don't see how it could possibly allow anywhere near the freedom you
get when passing around dynamic arrays that are slices of GC-allocated
memory. You end up throwing all kinds of complications into the mix just to
try to get a bit more @safety for non-GC-allocated memory. Maybe some of
those complications are worth it, but at some point, it's just too much
extra machinery for too little benefit, and I'd argue that if it harms
common use patterns of GC-allocated objects (which this DIP definitely
does), then it's going to far.

Personally, I don't think that it makes much sense to try to improve passing
around naked slices of malloc-ed memory. The lack of lifetime management
makes it so that you can't possibly guarantee that it's @safe to free the
memory without severely restricting what's allowed, and in general, I'd
strongly argue that in most cases, it's bad practice to pass around naked
slices of malloc-ed memory unless what the code is deing is restricted
enough that the programmer knows what's being done with that memory so that
they can know when it's safe to free it. Some use of features like scope may
help make it more manageable, but I think that it's folly to expect code to
do something like

auto arr = (cast(int*)malloc(length * int.sizeof))[0 .. length];

pass that memory around, and then later do

free(arr.ptr);

and have that be @safe. Outside of simple examples, to pass around malloc-ed
memory @safely, it needs to be wrapped in a type that manages its lifetime
(e.g. a smart pointer). Language improvements (like DIP 1000) may make it
easier to manage when pointers or references to that memory temporarily
escape the smart pointer in order to operate on it, but to pass around and
store that memory like you'd frequently do with GC-allocated memory, it
needs to be wrapped in something that manages its lifetime, just like you'd
do in languages like C++.

Regardless, even if all agreed that we wanted to improve the @safety of
malloc-ing a naked dynamic array, passing it around, and then freeing it, I
don't see how this DIP really helps with that. As has been pointed out
already in this thread, simply mutating the dynamic array itself screws with
your ability to get the pointer from it to free (showing that you really
should be storing a pointer for that separately regardless of what's going
on with appending), and appending doesn't cause any @safety problems. It
just makes it harder to track whether the dynamic array still refers to any
portion of that malloc-ed memory, but even then, it doesn't make it
impossible to do so, and if you want to forbid appending, you can simply
require that the code be @nogc. So, the DIP fails to even do what it's
trying to do, and in the process, it hurts very common use cases.

- Jonathan M Davis