Why do core.atomic functions require shared

[Jonathan M Davis]

On Wednesday, July 04, 2018 09:25:27 Boris-Barboris via Digitalmars-d wrote:
> Given the pain of shared usage with std and pretty much every
> library in existence, I cowboyed the server without this
> qualifier. One of the mechanisms required atomic class reference
> compare-and-set, and the class reference is not shared, because
> it would otherwise require, like, 30 or 40 casts to non-shared in
> other places co compile. I was then mortified to learn that
> core.atomic operate on shared references\pointers\values, and
> that I had to do stuff like:
> ```
> cas(cast(shared(C)*) &unshared_c, cast(shared C) unshared_c,
> cast(shared C) unshared_c);
> atomicStore(*(cast(shared(C)*) &unshared_c), cast(shared(C))
> unshared_c);
> ```
>
> Does shared impose some alignment constraints in the backend,
> that are needed for core.atomic? Do these functions really need
> to work only with shared?

It wouldn't make any sense for atomics to operate on anything other than
shared data. If the data isn't shared, then it doesn't need atomics.

There are some aspects of shared that still need to be fixed up (and Walter
and Andrei have been discussing it), but the basic idea with shared is that
you shouldn't be able to operate on it directly unless the compiler can
guarantee that what you're doing is thread-safe. There's some debate as to
whether that should involve the compiler inserting stuff for you to make
stuff thread-safe or mean that you can't do much of anything to shared
unless you protect the data and cast it away (Andrei tends to prefer the
former, whereas Walter prefers the latter, and it sounds like Andrei is
coming around to Walter's way of thnking, but we'll see). Most of the way
that shared currently works, it goes with not allowing stuff and does not
insert anything to make anything thread-safe, but that isn't fully
implemented, because some stuff like copying shared data still works even
though it's not thread-safe.

At this point, to operate on anything that's shared, either means using
atomics or protecting the data with a mutex (be that with a synchronized
block / function or a mutex object) and temporarily casting away shared
while operating on the data. Afterwards, the mutex is released, and at that
point, there should just be only shared references to the data. About the
only time that operating directly on a shared object then makes sense is
when it manages the atomics or mutex and associated cast internally.

Of course, the fact that you can't operate directly on shared can get
annoying, but it's a side effect of the fact that non-shared objects are
supposed to be guaranteed to be thread-local, and it prevents the programmer
from accidentally operating on shared data in a manner that isn't
thread-safe. So, by and large, it's preventing bugs - though we certainly
need to lock it down better and preferably find some ways to make it more
user-friendly where we can (synchronized classes are supposed to help with
that but have yet to be finished).

Unfortunately, we have done a poor job of messaging how shared is supposed
to be used, and many expect to be able to operate on shared data like normal
data like you'd do in languages like C++ or Java, and so they get frustrated
fast. The atomics and mutexes are exactly what you'd be doing in those
languages. It's just that they don't protect you against accidentally
operating on shared data and thus don't have shared in the type system and
don't require casts. If we can finish locking down shared in the spec and
implementation and then properly message how to use it, I expect that that
will fix many of the problems (though some aspects of it are bound to always
be annoying).

Many seem to just use __gshared as a way out, but it's really only intended
for interacting with C global variables (and even then, only very carefully
- especially if it's an aggregate type), and it results in shared data being
treated as thread-local by the compiler, which can result in subtle, nasty
bugs.

> Side question: how hard is it for a semantic analysis to
> implicitly remove all shared qualifiers from all symbols
> referenced inside "synchronized" block (including "this"
> pointer). It would solve about 90% of my gripes with shared, I
> believe.

It's a very difficult problem. Synchronized classes were proposed in an
attempt to solve it, but even if they were fully implemented, they'd only
help partially, because they can only strip off the outermost layer of
shared. In order for the compiler to cast away shared for you, it has to be
able to guarantee that there are no unprotected references to that data, and
because D doesn't have any kind of ownership system in the language, we
don't have a clean way to do it.

Given D's lack of ownership model, any solution that might work would almost
certainly have to have some way to associate a mutex with an object (or set
of objects) and prevent all operations on the object which could allow any
unprotected references to the data to escape. That's what synchronized
classes try to do, but they'd have to somehow be locked down even further in
order to guarantee that anything other than that the data directly in the
object is protected. As soon as you can do something like return an
unprotected pointer or reference from a member function, then the compiler
can't guarantee that the data is protected and thus can't implicitly remove
that layer of shared.

If anyone can come up with a clean and sane way to fully protect an object
with an associated mutex, then we could do better with implicitly removing
shared, but I don't know of any way that it could be done without either
locking down what you can do with such an object so thoroughly that it
borders on unusable or adding ownership semantics to the language, and
ownership semantics would complicate the language so much that I'd be
shocked if we ever had them. So, right now, it's looking like synchronized
classes are probably the best that we're going to get, though maybe someone
smart will come up with something.

- Jonathan M Davis