Discussion Thread: DIP 1028--Make @safe the Default--Final Review

[Steven Schveighoffer]

On 4/4/20 2:53 AM, Walter Bright wrote:
> On 4/3/2020 2:06 PM, Steven Schveighoffer wrote:
>> But what should absolutely not compile is:
>>
>> extern(C) int free(void *);
>>
>> void foo(int *ptr) // now inferred @safe
>> {
>>     free(ptr);
>> }
> 
> I understand your proposal. You want C functions without bodies to be 
> @system.

This is the bare minimum proposal. There are various options as I laid out.

My preference is that ALL extern(C), extern(C++), extern(Objective-c) 
functions are @system by default, with or without body. This is the 
least intrusive and most consistent proposal that does not gut all 
safety guarantees in D.

>> The fact that we cannot control where/how people define their 
>> prototypes means we have to be firm on this. They need to opt-in to 
>> @safe with extern(C), it cannot be default.
> 
> On the other hand, special cases like this tend to cause unexpected 
> problems in the future. Experience pretty much guarantees it. It's 
> likely to be tricky to implement as well.

This is like saying we shouldn't keep loaded guns in a locked safe 
because we don't keep our shirts in a locked safe, because it's too 
complicated and "tricky to implement". Yep, it's not as easy. That's 
because it's dangerous.

> People remember simple rules. They don't remember rules with odd 
> exceptions to them, that always winds up with trouble and bug reports. 
> Simple rules applied evenly lead to a compiler that works and is 
> reliable. I'm afraid the weight of all the special rules will crush D.

This is not a "special rule" or even a complicated one. It's really 
simple -- extern(C) functions cannot be trusted, so they need to be 
@system by default. H.S. Teoh put it perfectly:

> The rule:
> 
> 	extern(D) => @safe by default
> 	extern(C) => @system by default
> 
> hardly sounds "odd" to me.  It almost verbally describes what C is, and
> what we want D to be, there's nothing easier to remember.

Continuing...

> Now, as to what to do. I spent a few minutes and added `@system:` in to 
> the C headers in druntime for windows, posix, and linux. Done. I hope 
> someone else will do it for freebsd, etc., if not I'll go do that to.

I don't even have to look to know that you didn't get them all. There 
are peppered extern(C) prototypes all over Druntime and Phobos. I 
pointed out one in another post which you have not replied to. This does 
not even mention all D code in all repositories which frequently add an 
extern(C) prototype for functions needed. Our current documentation says 
[https://dlang.org/spec/interfaceToC.html#calling_c_functions]:

> C functions can be called directly from D. There is no need for wrapper functions, argument swizzling, and the C functions do not need to be put into a separate DLL.
> 
> The C function must be declared and given a calling convention, most likely the "C" calling convention, for example:
> 
> extern (C) int strcmp(const char* string1, const char* string2);
> and then it can be called within D code in the obvious way:
> import std.string;
> int myDfunction(char[] s)
> {
>     return strcmp(std.string.toStringz(s), "foo");
> }

Hey look, there's a safety violation right there! It doesn't say, 
"import core.stdc.string where I've helpfully already pre-marked your 
system functions for you" it says, just spit out a prototype (without a 
@system tag) and you are good to go.

Let's update that documentation, and then wait for the questions "why 
does it say to use @system?"

"Oh, that's because D decided to trust all C calls as perfectly memory 
safe, so it's on you to tell the compiler it's not safe. Make sure you 
do that."

WAT.

> 
>  > is going to cause huge issues.
> 
> I doubt that for the simple reason that @system by default has not 
> caused huge issues.

@system by default is fine! It doesn't violate safety because you have 
to opt-in to trusting code. @safe by default for code that CANNOT BE 
CHECKED is wrong. Just plain wrong, and breaks safety completely.

@safe by default for code that CAN be checked is fine, because the 
compiler will check it. If it's not safe, it won't compile. This is why 
it's ok to mark @safe by default D functions with implementation, and 
even extern(D) prototypes (since the name mangling takes into account 
safety).

> 
> The rule is simple:
> 
> "For a D module with a bunch of C declarations in it, start it with 
> `@system:`."

This is your "simpler" solution? "Don't use the default because it's 
completely wrong"

And that is not the only place C prototypes are declared. People who 
have a C function they need to call follow the spec, and add a prototype 
into their module that is full of D code. Putting @system: at the top 
isn't a good idea.

> 
> It's not a hard rule to check. It's one line. D's C interface has always 
> relied on the user to get right. Recall that C doesn't do any name 
> mangling at all:
> 
>    ----- mylib.di
>      extern (C) int addOne(int i);
> 
>    ----- mylib.c
>      double addOne(double d) { return d + 1; }
> 
> 
> That'll link fine and yet fail at runtime. Oops! Calling it @system will 
> not help at all. If the C implementation is bad, there's not a damn 
> thing D can do about it, with or without @system. It's always going to 
> be this way.

This is a red herring -- code that doesn't work is not code that I care 
about. ALL unmarked extern(C) prototypes in existence today THAT ARE 
CORRECTLY WRITTEN are @system!!! Making them "magically" switch to @safe 
is wrong, BUT THEY WILL STILL WORK. However D safety will be utterly 
gutted and destroyed. I can't say this enough.

Consider a scenario:

Before the switch:

Some library has code that does i/o. They use prototypes to interface 
with the system calls as that's what the spec calls for:

extern(C) size_t read(int fd, void* buf, size_t length);

size_t doRead(int fd, void[] arr)
{
    return read(int fd, &arr[0], arr.length);
}

Now, let's see what user code can do here:

void main()
{
    int[10] arr;
    doRead(0, arr[]);
}

This runs and builds, and is technically fine! But it's not @safe. Which 
is OK because the user didn't opt-in to safety. This ALSO compiles:

void main()
{
    int[][10] arr;
    doRead(0, arr[]);
}

This compiles and builds and is NOT fine. It's now reading POINTERS out 
of stdin. It's still not @safe, and the user did not declare it safe, so 
it's on him (maybe he knows what he's doing).

Now, let's move to the future where your new DIP is the default.

BOTH versions of user code COMPILE, and are treated as @safe!!! Why? 
simply because the read prototype is now considered @safe.

This SILENTLY still works, and the library function doRead now is 
incorrectly @safe. Perhaps the user knew what he was doing when he was 
reading pointers from stdin. Maybe it's OK for now, but the library 
DEFINITELY is wrong for other uses.

You see, the problem isn't that "someone didn't do it right", it's that 
the thing that was right before is now wrong. Instantly, code that was 
completely correct in terms of memory safety is now completely 
incorrect. And it still silently builds and runs exactly as it did before.

Changes like this should REQUIRE a deprecation and error period.

But the easier thing is to simply avoid marking things safe that aren't 
safe. With that one change, this whole DIP becomes a benefit rather than 
the great destructor of all safe code in existence.

Please reconsider your position. This is so important, I think a virtual 
live discussion is in order if you are not convinced by this. Let me 
know, I'm working from home for 3 weeks already, I'm fine having one any 
time.

-Steve