Manu's `shared` vs the @trusted promise

[Manu]

On Sun, Oct 21, 2018 at 3:05 PM ag0aep6g via Digitalmars-d
<digitalmars-d at puremagic.com> wrote:
>
> It took me a while to understand Manu's idea for `shared`, and I suspect
> that it was/is the same for others. At the same time, Manu seems
> bewildered about the objections. I'm going to try and summarize the
> situation. Maybe it can help advance the discussion.
>
>
> (1) How Does Manu's `shared` Interact with @trusted?
>
> With Manu's `shared`, there is implicit conversion from non-`shared` to
> `shared`. It would essentially become a language rule. For that rule to
> be sound, any access to `shared` data must be @system. And more
> challengingly, @system/@trusted code must be written carefully with the
> new rule in mind.

Well, just to be clear, I wouldn't say "access to shared data is
@system", I'd say, "there is no access to shared data".
The only way to access data members from a shared instance is to cast
away shared; and naturally, that is @system. So if this is what you
mean, then I agree.

> (Manu, you might say that the conversion follows from `shared` methods
> being "guaranteed threadsafe", but I think it's easier to reason this
> way. Anyway, potayto potahto.)
>
> The consequence is: In @trusted code, I have to make sure that I have
> exclusive access to any `shared` data that I use.

This isn't strictly true, other access are possible, as long as
they're atomic and you encapsulate properly, but I'll go with you
here...

> If code that is not
> under my control can obtain a non-`shared` view of the same data, I have
> failed and my @trusted code is invalid.

Right.

> An example in code (just rehashing code given by Manu):
>
> ----
> struct Atomic
> {
>      private int x;
>
>      void incr() shared @trusted
>      {
>          /* ... atomically increment x ... */

           atomicIncrement(cast(int)*&x); // <- I added the unsafe
call to atomicInc for clarity

>      }
>
>      /* If this next method is here, the one above is invalid. It's the
>      responsibility of the author of the @trusted code to make sure
>      that this doesn't happen. */
>
>      void badboy() @safe { ++x; } /* NO! BAD! NOT ALLOWED! */
> }
> ----

Right, this is the rule that I depend on. It carries the weight of the model.
Fortunately, the number of tools of this sort are very few in number,
and you probably would never write any of them yourself.


> (2) What's Wrong with That?
>
> The @trusted contract says that an @trusted function must be safe when
> called from an @safe function. That calling @safe function might be
> located in the same module, meaning it might have the same level of
> access as the @trusted function.
>
> That means, Atomic.incr is invalid. It's invalid whether Atomic.badboy
> exists or not. It's invalid because we can even possibly write an
> Atomic.badboy. That's my interpretation of the spec, at least.

It's @trusted, not @safe... so I don't think you can say "It's invalid
because we can even possibly write an Atomic.badboy" (I would agree to
that statement if it were @safe).
That's the thing about @trusted, you have to trust the engineer to
confirm contextual correctness.

But semantics aside, how and why did you add code to this module? Do
you usually add code to druntime or phobos?
Assuming you did add code to this module, are you telling me that you
don't understand what Atomic() does, and you also did not understand
the rules of `shared`? You can't be trusted to write threadsafe code
if you don't understand `shared`s rules.
My point is, do you genuinely believe this is a high-risk? When did
you last rewrite Atomic(T) because you didn't like the one in
druntime? Have you ever heard of a case of that?

I mean, I understand it's _possible_ to violate incr()'s promise, and
that's why it's @trusted, and not @safe. But what's the probability of
that happening by accident... and would you *honestly* make an
argument that this unlikely scenario is more likely to occur than any
of your 900 high-level engineers making any sort of mistake with
respect to the *use* of shared's current rules, which require unsafe
interaction at every call, by every end-user?
Most users don't modify druntime.

I think people are drastically over-estimating how much such @trusted
code would exist. Everybody fears multithreading, and nobody wants to
write data-races. If you lived in a world where we had a model to
describe @safe multithreading, you would do everything you can to stay
inside that playground, and engaging in unsafe threading code would
quickly become a pungent stench.
I suspect you only think about writing @trusted code so much because
that's what the existing implementation of `shared` has trained us all
to do.

> But according to Manu, Atomic.incr is fine as long as there's no
> Atomic.badbody that messes things up. So it looks like we're expected to
> violate the @trusted contract when dealing with Manu's `shared`. But
> when we routinely have to break the rules, then that's a sign that the
> rules are bad.

I lost you here... how are you "expected" to violate the contract...
and "routinely break the rules"?
I'm telling you to NEVER do that. How can you say it's expected, or
that it's routine?

There are like, 2 times I can think of:
1. When the person that writes Atomic(T) writes it. I'll do it if you
like, and I won't bugger it up.
2. When some smart cookie writes LockFreeQueue(T), and everyone uses
it, because nobody would dare ever write that class themselves.
That's almost all the times that you would ever see code like this.

There is also Mutex, and Semaphore, but they're trivial.

These modules that you refer to should be very small and tight.
Threadsafe promises are encapsulated by the module.
Threadsafety is hard, and you do not write a crap-load of code and
hope it's all correct. You write the smallest amount of code possible,
then you test it.
You DON'T go and start messing with foundational threadsafe code,
because you're not a moron.

I agree the situation you fear is technically possible, but I think
it's very unlikely, and in balance to the risks associated with shared
today, which is completely unsafe at the user-facing level, and also
completely unregulated (you can access members freely)...

> (3) Maybe It Can Be Made to Work?
>
> There might be a way to implement Atomic without breaking the @trusted
> promise:
>
> ----
> struct Atomic
> {
>      shared/*!*/ int x;
>
>      void incr() shared @trusted { /* ... */ }
>
>      /* Now this gets rejected by the compiler: */
>      void badboy() @safe { ++x; } /* compiler error  */
> }
> ----

Yeah, that's probably fine.

> With a `shared int x` there's no way that @safe code might access it, so
> the @trusted promise is kept.
>
> Manu, I don't know if marking fields like this is compatible with your
> plans. But it would address the @safe-ty issue, I think.

Yeah, that's fine granted my rules.
Another option might be to wrap the volatile member like `x` here in a
`shared` property, and only access it through that.
Perhaps it's possible that the compiler could note when a data member
is accessed within a shared function, and then make a noise any time
that same symbol is accessed directly in a non-shared function in the
same module (recommending to use a `shared` property).

I've said on at least 5 other occasions, I'm sure there are a whole
lot of options we can explore to assist and improve the probability
that the ground-level author doesn't make a mistake (although I am
quite confident you'd be wasting your time, because they won't make a
mistake like this anyway).
That conversation has nothing to do with the validity of the rules
though, which is what the other 400 post thread is about.

If you *do* trust the 5-10 @trusted functions in the library; Is my
scheme sound?
If the answer is yes, then we can talk about how to improve our odds.

> However, even if it's possible to reconcile Manu's `shared` with @safe
> and @trusted, that doesn't mean it's automatically golden, of course. It
> would be an enormous breaking change that should be well thought-out,
> scrutinized, planned, and executed.

Sure. But the OP, and every one of the 400 posts later, are trying to
determine if the scheme is sound.
We can fuss about the details until the cows come home, but I'm
finding it impossible to get everyone on the same page in the first
place.