Very limited shared promotion

[Jonathan M Davis]

On Tuesday, June 18, 2019 3:42:50 PM MDT Ola Fosheim Grostad via 
Digitalmars-d wrote:
> On Tuesday, 18 June 2019 at 21:20:13 UTC, Jonathan M Davis wrote:
> > escape. However, all that's required is a piece of @trusted
> > code with access to the scope object, and all protections go
> > out the window. At that point, it's up to the programmer to
>
> So if I implement a container with @trusted method that grabs a
> reference to the object, which is reasonable, then it will fail?

Storing a reference to a scope object in @trusted code would violate what
scope is supposed to guarantee. So, such code would be badly written code
just like @trusted code that screws up bounds-checking with pointer
arithmetic is bad code. Temporarily taking the address of a scope object and
doing something with it but making sure that no such references exist when
the function exits could be reasonable in @trusted code, because the
programmer would have ensured that the guarantees that scope is supposed to
provide would have been met. So, you _can_ do stuff that scope doesn't allow
which doesn't actually violate scope's guarantees, but if your @trusted code
does anything which violates the guarantees that @safe is supposed to make,
then the programmer who verified that it was okay to mark it as @trusted
screwed up.

> Or is there a "temporarily unshared" type that the container
> implementor could test for? Or is the argument that this would
> also fail for new/delete of reference counting so that you will
> try to reuse all the ref counting semantics on the shared level?
>
> Is the idea to treat a grabbed lock as "generating a new object"
> and deal with it as refernce counting? Then you need to check
> that all locks have been released to avoid deadlocks/starvation...
>
> This will only work in simple scenarios though...

@safe really only deals with memory safety, not thread-safety. Converting
between shared and thread-local does require a cast, which is @trusted,
because you're basically stepping outside of the type system. At that point,
the code involved has to then deal with the threading stuff properly, but as
far as the type system is concerned, an object is either thread-local or
shared, and you're on your own if you use casts to change something from one
to the other even temporarily. @safe is only involved insofar as the cast is
@system.

What Manu is proposing is a scenario where the type system is able to
guarantee that no references to the variable escape and that based on that
assumption, temporarily converting to shared wouldn't violate the guarantees
that come with the object actually being thread-local. That _does_ rely on
the code then either all being @safe or that any @trusted bits are correctly
vetted by the programmer to ensure that no references to the scope object
actually escaped. If the programmer screws that up, then the implicit
conversion to shared will have violated the guarantees that are supposed to
go with the object being thread-local, and unlike now, the point of the
conversion wouldn't be @trusted, because it would be done implicitly by the
compiler based on the assumption that scope and @safe were properly upheld
within the function being called (which is true of any @safe function
involving scope except that normally that doesn't involve any conversions
to or from shared unless the programmer does something @system to make it
happen).

On the surface, what Manu is proposing _seems_ sound (assuming that any
@trusted code involved is vetted properly), but as Walter points out, it's
really easy to screw up threading stuff. And personally, I'm inclined to
argue that we're better off having the point where a variable is converted
to or from shared always be @system so that it's never invisible, since
pretty much anything involving shared needs to be vetted. Also, normally,
shared objects either deal with their thread-safety stuff internally (e.g.
by having an internal mutex that locks appropriately when accessing the
object's members), or they require that you deal with the thread
synchronization stuff explicitly (e.g. by directly dealing with the mutex
whenever the shared object needs to be accessed). Having a function that's
expecting a shared variable be given a thread-local one seems off to me.
That's basically what's required when passing objects across threads, but at
that point, scope wouldn't be involved, and it would definitely be up to the
programmer to make sure that no other references to the object exist before
casting it to shared to pass it across threads. It could very well be that
Manu has found a totally reasonable use case where it makes sense to
temporarily convert a thread-local object to shared without letting it
escape, but I'm still inclined to think that the conversion should be vetted
by the programmer rather than being considered okay and done implicitly just
because scope is involved.

Regardless, as discussed at dconf this year, D's memory model and the exact
semantics of shared really need to be properly locked down before we start
making changes like this. We know basically what shared is and how it works,
and we even have some idea of some of the changes that need to be made (e.g.
it's almost certainly the case that reading and writing shared variables
needs to become illegal, thus requiring casts to do either, because neither
is thread-safe without thread synchronization mechanisms that the programmer
needs to put into place and which the compiler doesn't understand well
enough to do any conversions for you), but not all of the details have been
properly ironed out yet, and the devil is in the details.

- Jonathan M Davis