Very limited shared promotion

[Ola Fosheim Grøstad]

On Tuesday, 18 June 2019 at 22:30:28 UTC, Jonathan M Davis wrote:
> Storing a reference to a scope object in @trusted code would 
> violate what scope is supposed to guarantee.

Ok, so you are only considering stack allocated objects?

My question is more like this. If I use GC throughout, then I 
might write a @trusted framework with that in mind, and take 
references wherever it makes sense. No problem, right?

However, when someone pass a shared (still GC managed) object 
that has been temporarily "unshared", and pass it into that to 
the framework... then it will break badly.

So, from a memory-management perspective the framework is sound, 
but not from a thread safety point of view.

Is that right?


> but if your @trusted code does anything which violates the 
> guarantees that @safe is supposed to make, then the programmer 
> who verified that it was okay to mark it as @trusted screwed up.

But only if it was vetted with "shared" in mind, because if it 
was vetted for safe release of memory, and you use GC or 
reference counting, then it was perfectly OK, but still breaks 
for "unshared" shared?

Maybe this will demand too much from the person doing the vetting 
of generic code, to both get it right for memory management and 
thread safety?

> @safe really only deals with memory safety, not thread-safety. 
> Converting between shared and thread-local does require a cast, 
> which is @trusted, because you're basically stepping outside of 
> the type system.

Ok, but what if you didn't step outside the type system? What 
would D need to cover this within the type system?

> What Manu is proposing is a scenario where the type system is 
> able to guarantee that no references to the variable escape and 
> that based on that assumption, temporarily converting to shared 
> wouldn't violate the guarantees that come with the object 
> actually being thread-local.

That sounds reasonable.

> scope object actually escaped. If the programmer screws that 
> up, then the implicit conversion to shared will have violated 
> the guarantees that are supposed to go with the object being 
> thread-local, and unlike now, the point of the conversion

Yes, but this is where a more elaborate type system would help. I 
haven't mentioned Pony
( https://www.ponylang.io/ ) in a while, so to recap:

Pony tracks whether an object-reference is unique among other 
things.

https://tutorial.ponylang.io/reference-capabilities/reference-capabilities.html

So, that might be worth considering.

I don't think it has to be tedious if you use auto/type 
deduction/flow typing.

> On the surface, what Manu is proposing _seems_ sound (assuming 
> that any @trusted code involved is vetted properly), but as 
> Walter points out, it's really easy to screw up threading stuff.

Yes, that is true, and "shared" isn't even the most challenging 
problem. The real challenge is to prove that starvation/deadlock 
cannot happen in a complex system.

(which of course is a good reason to avoid complex systems in the 
first place)

> Also, normally, shared objects either deal with their 
> thread-safety stuff internally (e.g. by having an internal 
> mutex that locks appropriately when accessing the object's 
> members),

Only works in very simple scenarios, you often need to grab 
multiple resources at once. Also, fine granularity locks tend to 
come with high performance penalties.

So, I think one should exclude that solution and focus on the 
general case. Single object sync isn't really a solution that 
will convince anyone that a language has nailed concurrency…

> or they require that you deal with the thread synchronization 
> stuff explicitly (e.g. by directly dealing with the mutex 
> whenever the shared object needs to be accessed).

Yes, I agree if you change "object" to "objects".

> Having a function that's expecting a shared variable be given a 
> thread-local one seems off to me.

Well, if you use reference-capabilites in the vein of Pony then I 
think it follows naturally.

And that appears to be what D is trying to do with "shared", but 
without the precision needed to solve issues that seems 
reasonable to people working in the trenches (like Manu).

> shared without letting it escape, but I'm still inclined to 
> think that the conversion should be vetted by the programmer 
> rather than being considered okay and done implicitly just 
> because scope is involved.

But, to play the devil's advocate:

If the compiler doesn't provide strong semantic passes for 
"shared" why do you then need "shared" to be part of the type 
system?  Why couldn't you then just let "shared" be implemented 
as a template within a templated pointer framework?

For "shared" to be justified as a language feature it has to 
provide something that cannot be done within the meta-programming 
capabilities of the language.

For "const" it is obvious, the transitive const cannot be done 
within the meta-programming capabilities of the language (or 
maybe it can, I am making an assumption).

But how does "shared" justify itself?  Is it all about 
@safe/@trusted/@system?

In that case, maybe those capabilities could be available as 
meta-programming mechanisms so that "shared" could be done as a 
library feature.

Or rather, what prevents "shared" from being a library feature 
(assuming all pointers are templates).

> Regardless, as discussed at dconf this year, D's memory model 
> and the exact semantics of shared really need to be properly 
> locked down before we start making changes like this.

Indeed.

> but not all of the details have been properly ironed out yet, 
> and the devil is in the details.

Certainly.

I would also argue that whatever you land on for managing a 
resource like memory, also should be considered to be extended to 
manage a "resource" like read/write access to shared objects.

Ola.