synchronized - shared but actually useful

[Jonathan M Davis]

On Wednesday, October 31, 2018 1:00:32 AM MDT FeepingCreature via 
Digitalmars-d wrote:
> On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis
>
> wrote:
> > But shared is a low level construct and should stay that way.
> > It's the base for everything else.
>
> This is simply not correct. People didn't stop having a need for
> threading while D was figuring out how to unbreak shared for ten
> years and counting. Fact is, the base for everything else is
> __gshared and synchronized, and it will *continue* to be
> __gshared and synchronized unless __gshared is removed and the
> Thread constructor is changed to only take shared delegates.
>
> Why not do something useful with synchronized instead?

__gshared is intended for interacting with C globals and that's it. If it's
used for anything else, it's not being used for its intended purpose.
Certainly, using it for D objects is just begging for bugs. __gshared should
_never_ be used for D objects. I know that some folks do, and sometimes
folks get away with it, but it's just asking for trouble, and there's no
guarantee that such code will continue to work, because it's violating
compiler guarantees when it does so. __gshared is designed solely as a
backdoor for dealing with C and that is it. On the whole, shared really is
_not_ broken and _does_ work. Yes, it has some rough edges that need fixing,
and not everything in core.sync has been updated to use it properly,
necessitating more casts than should be necessary when using those
constructs than should be necessary, but shared as a whole is sound and
always has been. The biggest problem with it is that most folks don't
understand how to use it properly.

> > And to be honest, I don't really expect it to be reasonable to
> > add much in the way of higher level mechanisms for sharing data
> > to D at the language level - not with it being a systems
> > language. Most stuff would require restricting it further
> > and/or introducing a serious ownership model to it, which we
> > really don't want to do. Even synchronized classes are pretty
> > questionable, because they can only safely remove the outer
> > layer of shared, which is arguably better than nothing, but it
> > doesn't get you very far. Without a full-blown ownership model,
> > we're pretty much reduced to manual casting when mutexes or
> > synchronized are used, and the language really isn't going to
> > be much help. If someone can come up with some bright ideas,
> > great. All the more power to them. Maybe we can improve the
> > situation. But again, they should be built on top of shared as
> > a low level mechanism, not try to change what shared is.
>
> Sorry, but can you please actually comment on my proposal in
> concrete terms rather than handwave about how we're reduced to
> manual casting (which is not necessary for concurrency at the
> moment, so I'm not sure why you think it's unavoidable), and how
> the language surely cannot be of much help without even
> commenting on a proposal for how it *could* actually be of help?

Manual casting is required if you're actually using shared with mutexes or
synchronized. If you're using __gshared, it's not, but that's only because
you're lying to the compiler about whether the variables are shared are not,
which means that you're circumventing the type system, and you're risking
subtle bugs in your programs, because the guarantees that the type system is
suppose make aren't actually in place.

Your synchronized field proposal is basically just a variation of
synchronized classes with the extra caveat that somehow mutable references
to the member variables can't escape, and for some reason, not everything in
the class has to be synchronized. It's like you're trying to have
synchronized classes without talking about shared. But you can't take shared
out of the mix. Per the type system, anything that isn't shared is
considered thread-local by the compiler, and trying to do anything like this
without shared would be a serious violation of the type system and the
compiler's guarantees.

Instead of talking about safely casting away the outer layer of shared
inside the class, you're talking about preventing escaping. synchronized
classes as described in TDPL already guaranteed that the outer layer can't
escape, since it lives directly in the class. What they can't guarantee is
that the rest can't escape, and the language doesn't provide any way for
such a guarantee. scope (even with DIP 1000) isn't transitive, so it doesn't
really do the trick. So, I don't know how we could guarantee that _nothing_
escapes, though if nothing escapes, then all levels of shared could be
implicitly cast away instead of just the outer layer, which would be nice.

Also, even if we _could_ somehow prevent anything from escaping, I would
point out that mutability has nothing to do with this (or at least constness
doesn't), because reading has many of the same threading issues as writing.
In order to avoid threading issues, the data must actually be immutable. So,
allowing const references to escape would still violate thread-safety.
Access to data must be synchronized unless it is immutable - whether we're
talking about reading or writing is irrelevant.

synchronized classes work (or would work if fully implemented) by
guaranteeing that a single level can't escape - the one level that it _can_
guarantee can't escape - and thus are able to properly synchronize that
level. If scope were transitive, then maybe we could do better, but since it
isn't (and Walter insists that it can't reasonbly be transitive), I'm not
sure that it's really reasonable to prevent escaping transitively.

In any case, your proposal is just a variation of synchronized classes. So,
while it may state it in a new way, the basic concept really isn't anything
new. And my point still stands that anything that we do with regards to
thread safety must be built on top of shared, because that is the building
block of sharing data across threads in D. Whether you like the specifics of
shared or not, the idea that the compiler can rely on the assumption that
data is _not_ shared across threads unless it's marked as such is a key
feature of D. And attempting to use __gshared to get around it is just
shooting yourself in the foot.

- Jonathan M Davis