synchronized - shared but actually useful

[FeepingCreature]

On Wednesday, 31 October 2018 at 07:57:59 UTC, Jonathan M Davis 
wrote:
> On Wednesday, October 31, 2018 1:00:32 AM MDT FeepingCreature 
> via Digitalmars-d wrote:
>> On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis
>>
>> wrote:
>> > But shared is a low level construct and should stay that 
>> > way. It's the base for everything else.
>>
>> This is simply not correct. People didn't stop having a need 
>> for threading while D was figuring out how to unbreak shared 
>> for ten years and counting. Fact is, the base for everything 
>> else is __gshared and synchronized, and it will *continue* to 
>> be __gshared and synchronized unless __gshared is removed and 
>> the Thread constructor is changed to only take shared 
>> delegates.
>>
>> Why not do something useful with synchronized instead?
>
> __gshared is intended for interacting with C globals and that's 
> it. If it's used for anything else, it's not being used for its 
> intended purpose. Certainly, using it for D objects is just 
> begging for bugs. __gshared should _never_ be used for D 
> objects. I know that some folks do, and sometimes folks get 
> away with it, but it's just asking for trouble, and there's no 
> guarantee that such code will continue to work, because it's 
> violating compiler guarantees when it does so. __gshared is 
> designed solely as a backdoor for dealing with C and that is 
> it. On the whole, shared really is _not_ broken and _does_ 
> work. Yes, it has some rough edges that need fixing, and not 
> everything in core.sync has been updated to use it properly, 
> necessitating more casts than should be necessary when using 
> those constructs than should be necessary, but shared as a 
> whole is sound and always has been. The biggest problem with it 
> is that most folks don't understand how to use it properly.
>
>> > And to be honest, I don't really expect it to be reasonable 
>> > to add much in the way of higher level mechanisms for 
>> > sharing data to D at the language level - not with it being 
>> > a systems language. Most stuff would require restricting it 
>> > further and/or introducing a serious ownership model to it, 
>> > which we really don't want to do. Even synchronized classes 
>> > are pretty questionable, because they can only safely remove 
>> > the outer layer of shared, which is arguably better than 
>> > nothing, but it doesn't get you very far. Without a 
>> > full-blown ownership model, we're pretty much reduced to 
>> > manual casting when mutexes or synchronized are used, and 
>> > the language really isn't going to be much help. If someone 
>> > can come up with some bright ideas, great. All the more 
>> > power to them. Maybe we can improve the situation. But 
>> > again, they should be built on top of shared as a low level 
>> > mechanism, not try to change what shared is.
>>
>> Sorry, but can you please actually comment on my proposal in 
>> concrete terms rather than handwave about how we're reduced to 
>> manual casting (which is not necessary for concurrency at the 
>> moment, so I'm not sure why you think it's unavoidable), and 
>> how the language surely cannot be of much help without even 
>> commenting on a proposal for how it *could* actually be of 
>> help?
>
> Manual casting is required if you're actually using shared with 
> mutexes or synchronized. If you're using __gshared, it's not, 
> but that's only because you're lying to the compiler about 
> whether the variables are shared are not, which means that 
> you're circumventing the type system, and you're risking subtle 
> bugs in your programs, because the guarantees that the type 
> system is suppose make aren't actually in place.
>
This does not match what I'm actually seeing in practice. If this 
is how it is, then this admonishment should be written in the D 
style guide in big, bolded letters.
Generally we try to write cast-free code if possible, so casting 
away shared feels wrong. That's not really an argument, of 
course, but to me semantic casting indicates "no compiler, you 
are mistaken about the properties of this data." The compiler 
should never *require* casting.

> Your synchronized field proposal is basically just a variation 
> of synchronized classes with the extra caveat that somehow 
> mutable references to the member variables can't escape, and 
> for some reason, not everything in the class has to be 
> synchronized.

Everything in the class has to be synchronized. I'm not sure 
where I said differently.

> It's like you're trying to have synchronized classes without 
> talking about shared. But you can't take shared out of the mix. 
> Per the type system, anything that isn't shared is considered 
> thread-local by the compiler, and trying to do anything like 
> this without shared would be a serious violation of the type 
> system and the compiler's guarantees.
>

Again, imo this belongs in either the spec or the DStyle guide in 
big bold letters.

> Instead of talking about safely casting away the outer layer of 
> shared inside the class, you're talking about preventing 
> escaping. synchronized classes as described in TDPL already 
> guaranteed that the outer layer can't escape, since it lives 
> directly in the class. What they can't guarantee is that the 
> rest can't escape, and the language doesn't provide any way for 
> such a guarantee.

Not sure what you mean by "outer layer" here or this entire 
paragraph.

> scope (even with DIP 1000) isn't transitive, so it doesn't 
> really do the trick. So, I don't know how we could guarantee 
> that _nothing_ escapes, though if nothing escapes, then all 
> levels of shared could be implicitly cast away instead of just 
> the outer layer, which would be nice.
>
> Also, even if we _could_ somehow prevent anything from 
> escaping, I would point out that mutability has nothing to do 
> with this (or at least constness doesn't), because reading has 
> many of the same threading issues as writing. In order to avoid 
> threading issues, the data must actually be immutable. So, 
> allowing const references to escape would still violate 
> thread-safety. Access to data must be synchronized unless it is 
> immutable - whether we're talking about reading or writing is 
> irrelevant.

Indeed, const is useless for threading, which is why my proposal 
explicitly only notes immutable data as implicitly synchronized. 
We're on the same page.

> synchronized classes work (or would work if fully implemented) 
> by guaranteeing that a single level can't escape - the one 
> level that it _can_ guarantee can't escape - and thus are able 
> to properly synchronize that level. If scope were transitive, 
> then maybe we could do better, but since it isn't (and Walter 
> insists that it can't reasonbly be transitive), I'm not sure 
> that it's really reasonable to prevent escaping transitively.
>

The point of the proposal is not precisely to prevent escaping, 
but to prevent the escape of *data structures/references that 
offer the ability to mutate data unprotected.* Remember that in 
this model, the object is the unit of data ownership. So the two 
things that are allowed to escape are immutable data (and 
non-reference copied data), and synchronized classes, because 
they can guarantee that they're taking care of protecting their 
data.

> In any case, your proposal is just a variation of synchronized 
> classes. So, while it may state it in a new way, the basic 
> concept really isn't anything new.

I didn't say it was! I said it was *useful* and *straightforward*.

> And my point still stands that anything that we do with regards 
> to thread safety must be built on top of shared, because that 
> is the building block of sharing data across threads in D. 
> Whether you like the specifics of shared or not, the idea that 
> the compiler can rely on the assumption that data is _not_ 
> shared across threads unless it's marked as such is a key 
> feature of D. And attempting to use __gshared to get around it 
> is just shooting yourself in the foot.
>
> - Jonathan M Davis


And yet, there's no way to even *state* that the Thread 
constructor must not take a delegate that can access unshared 
data, and none of the current proposals list one. Should we 
demand that only shared data can be passed to new threads? That's 
going to limit their usefulness. With synchronized, we can 
deprecate the delegate constructor, then pass the thread data as 
synchronized objects to a pure function, thereby actually 
guaranteeing thread safety.