Proposal to make "shared" (more) useful

[Arafel]

On 09/13/2018 09:49 PM, Jonathan M Davis wrote:
> 
> Have you read the concurrency chapter in The D Programming Language by
> Andrei? It sounds like you're trying to describe something vere similar to
> the synchronized classes from TDPL (which have never been fully implemented
> in the language). They would make it so that you had a class with shared
> members but where the outer layer of shared was stripped away inside member
> functions, because the compiler is able to guarantee that they don't escape
> (though it can only guarantee that for the outer layer). Every member
> function is synchronized and no direct access to the member variables
> outside of the class (even in the same module) is allowed. It would make
> shared easier to use in those cases where it makes sense to wrapped
> everything protected by a mutex in a class (though since it can only safely
> strip away the outer layer of shared, it's more limited than would be nice,
> and there are plenty of cases where it doesn't make sense to stuff something
> in a class just to use it as shared).
> 

I hadn't read the book, but that's indeed the gist of what I'm 
proposing. I think it could be enough to restrict it to value types, 
where it's easier to assume (and even check) that there are no external 
references.

> 
> [snip]
> 
> If we're going to find ways to make shared require less manual work, it
> means finding a way to protect a shared object (or group of shared objects)
> with a mutex in a way that is able to guarantee that when you operate on the
> data, it's protected by that mutex and that no reference to that data has
> escaped. TDPL's synchronized classes are one attempt to do that, but the
> requirement that no references escape (so that shared can safely be cast
> away) makes it so that only the outer layer of shared can be cast away, and
> it's extremely difficult to do better than that with having holes such that
> it isn't actually guaranteed to be thread-safe when shared is cast away.
> Maybe someone will come up with something that will work, but I wouldn't bet
> on it. Either way, I don't see how any solution is going to be acceptable
> which does not actually guarantee thread-safety, because it would be
> violating the guarantees of shared otherwise. A programmer can choose to
> cast away shared in an unsafe manner (or use __gshared) and rely on their
> ability to ensure that the code is thread-safe rather than letting shared do
> its job, but that's not the sort of thing that we're going to do with a
> language construct, and given that the compiler assumes that anything that
> isn't shared or immutable is thread-local, it's very much a risky thing to
> do.
> 

I completely agree with this argument, however please note that there 
must be a sensible way to work with shared, otherwise we enter in the 
"the perfect is the enemy of the good" area.

For reference types it's somehow workable, because you can just cast 
away and store it in a new variable:

```
class A {
	this() { }
}

shared synchronized class B {
	this(A a) {
		a_ = cast (shared) new A; // no shared this()
	}
	void foo() {
		A a = cast () a_;
		// Work with it
	}
	private:
	A a_;
}
```

It's still somewhat cumbersome, specially if you have many such members, 
but still doable.

However, this is not possible for value types, and it makes it nigh on 
impossible to work with them in a sensible way. You have either to use 
pointers, or cast away every type you want to use it. None of them are 
what I would call "practical".

While not the biggest problem (see the later point), I still think that 
synchronized classes are a good compromise, specially with the 
restriction of only applying to full value types (no internal references 
allowed). Of course it is still perhaps possible to bypass that 
mechanism, but so is the case with many other ones (assumeUnique?).

If it's hard enough to do by mistake, it can be assumed that the people 
messing with it should know what they are doing.

Finally, you suggest using __gshared, and I'm not sure you're not having 
the same misunderstanding I had: __gshared implies "static", so it's not 
a valid solution for class fields in most cases.

> As for __gshared, it's intended specifically for C globals, and using it for
> anything else is just begging for bugs. Because the compiler assumes that
> anything which is not marked as shared or immutable is thread-local, having
> such an object actually be able to be mutated by another thread risks subtle
> bugs of the sort that shared was supposed to prevent in the first place.
> Unfortunately, due to some of the difficulties in using shared and some of
> the misunderstandings about it, a number of folks have just used __gshared
> instead of shared, but once you do that, you're risking subtle bugs, because
> that's not at all what __gshared is intended for. If you're using __gshared
> for anything other than a C global, it's arguably a bug. Certainly, it's a
> risky proposition.
> 

As I said, the current semantics of __gshared doesn't allow it to be a 
"drop-in" replacement of "shared". I also agree that it's not what it 
was meant for, and that changing that right now would risk breaking a 
lot of code.

However, I think that there should be *some* way in the language itself 
to express that without having to cast all over the place: access this 
member of the shared class as if it were local, I'll take care of 
controlling the access to it.

You can use a wrapper type, and that's what I'm trying to do right now, 
but I'm pretty sure there will be a ton of corner cases and interactions 
that will make it really hard to work reliably in a generic way.

Then, implementing a kind of "synchronized classes" that would automate 
this when possible would of course be a further and welcome improvement.

A.