Generality creep

[Jonathan M Davis]

On Saturday, March 30, 2019 6:41:15 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:
> On 3/30/19 2:49 PM, Jonathan M Davis wrote:
> > This would also be a great opportunity to fix some of the issues with
> > shared in druntime
>
> The problem here is, of course, that nobody knows exactly what shared is
> supposed to do. Not even me. Not even Walter.
>
> One monumental piece of work would be DIP 1021 "Define the semantics of
> shared". Then people who build with -dip1021 know what to do and what to
> expect. Now that would patch one of the bigger holes mentioned in the
> other post.

I confess that I find the amount of confusion over shared to be confusing,
though maybe I'm missing something. Yes, it has some details that need to be
worked out, but I would have thought that basics would be well understood by
now. For shared to work, it has to prevent code that isn't thread-safe -
which means either making any operation which isn't guaranteed to be
thread-safe illegal and/or making the compiler insert code that guarantees
thread-safety. The latter is very hard if not impossible, if nothing else,
because that would require that the compiler actually understand the
threading and synchronization mechanisms being used in a given situation
(e.g. associating a mutex with a shared variable and then somehow
guaranteeing that it's always locked at the appropriate time and unlocked at
the appropriate time). So, basically, that means that any operations that
aren't guaranted to be thread-safe then need to be illegal for shared, and
then to actually read or modify a shared object, you have to either use
atomics, or you have to use whatever synchronization mechanisms that you
want to be using (e.g. mutexes), cast away shared while the object is
protected, operate on the object as thread-local, make sure that no
thread-local references to it exist when you're done, and then release the
mutex. It's exactly what you'd be doing in C++ code except that you have to
worry about casting away shared while actually operating on the object,
because the type system is preventing you from shooting yourself in the foot
by reading or writing to shared objects, since that's not thread-safe
without synchronization mechanisms that aren't part of the type system.

It seems to be confusing for many people, because they expect to actually be
able to directly operate on shared objects, but you can't do that and
guarantee any kind of thread-safety unless you're calling stuff that takes
care of the protection for you (e.g. a type could contain a mutex and lock
it in its member functions, thereby encapsulating all of the casting away of
shared - which is basically what synchronized classes were supposed to do,
just with the outer level only, whereas you can do more than that if you're
doing it manually and write the code correctly). But shared types in general
aren't necessarily going to have any kind of synchronization mechanism built
in (e.g. shared(int*) certainly won't).

The primary problem that I see with shared as things stand is that it still
allows operations which aren't guaranteed to be thread-safe (e.g. copying).
It may need some tweaks beyond that, but as far as I can tell, it mostly
works as-is. However, it's much harder to use than it should be because the
stuff in core.sync has only been partially made to work properly with
shared.

I expect that when you get down to all of the nitty gritty details of making
operations that aren't guaranteed to be thread-safe illegal that there could
be some hairy things that need to be sorted out, but we've already made
_some_ of them illegal, and the addition of copy constructors to the
language actually fixes one of the big hurdles, which is making it possible
to make it thread-safe to copy an object (by having its copy constructor be
shared and handle the mutex or atomics or whatnot internally). So, by no
means am I claiming that we have it all figured out, but I would have
thought that it would primarily be an issue of figuring out how to correctly
make operations that aren't guaranteed to be thread-safe illegal (just like
finishing @safe by making operations @system when the compiler can't
guarantee that they're memory safe). Some of the details could turn out to
be nasty, but I would have thought that it would just be a matter of working
through them and that as things stand, the basic design of shared works.

Maybe I'm missing something here, but it seems to me that it _should_ be
pretty straightforward to just move forward from the idea that operations on
shared objects have to be thread-safe or be illegal. That may involve more
casting than would be ideal, but I don't see how we can really do much with
the language understanding enough to do the casting for you, and the result
is basically what you get in C++ - except that the compiler is preventing
you from shooting yourself in the foot outside of @system code where you're
handling the casting away of shared. So, it's pretty much the same situation
as @safe vs @system in the sense that shared code disallows operations that
the compiler can't guarantee the safety of, and you have to do @system stuff
to guarantee the safety yourself in the cases where you need to do that
stuff.

Regardless of the details of what we want to do with shared though,
core.sync doesn't handle it properly. Mostly, it doesn't use shared at all,
and when it does, it's hacked on. So, at some point here, we really need to
replace what's there with a v2 of some sort. And until we do, no matter how
well shared works on its own, it's going to be seriously hampered, because
the constructs in core.sync are core to writing threaded code.

- Jonathan M Davis