Shared

[Dominikus Dittes Scherkl]

On Tuesday, 14 May 2019 at 21:31:57 UTC, Jonathan M Davis wrote:
> On Tuesday, May 14, 2019 6:22:20 AM MDT Dominikus Dittes 
> Scherkl via Digitalmars-d wrote:
>> On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
>> > On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes 
>> > Scherkl
>> >
>> > via Digitalmars-d wrote:
>> >> On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis
>> >>
>> >> wrote:
>> >> > All that really would do is add a bit of extra syntax 
>> >> > around locking a mutex and casting away shared.
>> >>
>> >> No, it makes it possible to use mutex in safe functions 
>> >> without needing to cast and which the compiler CAN 
>> >> guarantee to really be safe.
>> >
>> > Actually, it doesn't. All you've done is lock a mutex and 
>> > cast away shared. There is no guarantee that that mutex is 
>> > always used when that object is accessed. There could easily 
>> > be another piece of code somewhere that casts away shared 
>> > without locking anything at the same time.
>>
>> But that piece of code is system, which explicitly allows you 
>> to shoot into your foot. If you want to stay safe, don't do 
>> that.
>
> The point is that if that code can legally exist, then the 
> compiler simply cannot guarantee that removing shared from the 
> object is thread-safe even with the locking mechanism you're 
> proposing.

with system code you can always destroy safety assumptions of any 
other written code. This is why system code should be avoided 
where ever possible and the unavoidable remains need to be 
reviewed very carefully to not spoil the guarantees that are 
valid otherwise.

>
>> > And even if this were the only mechanism for removing 
>> > shared, you could easily use it with the same object and a 
>> > completely different mutex in another piece of code
>>
>> Yes, the lock block need a list of vars that it allows to be 
>> modified
>>
>> lock(var1, var2, ...)
>> {
>> }
>>
>> two mutexes can only be executed at parallel if their 
>> parameter set is disjunct.
>
> Sure, but another thread could be using a completely different 
> mutex with one or more of those variables.

No, it can't. Disjunct means: It cannot be called unless all of 
the given variables are free (not locked by any other mutex).

>> ok, so we need in addition that a reference to a shared var 
>> need not be lived beyond the end of the locked block or be 
>> immutable. Bad, but seems necessary.
>
> A reference could already exist before your proposed locking 
> mechanism was reached in the code. If the type is a class or 
> pointer, then there could be other class references or pointers 
> to the same data in @safe code. And in @system/@trusted code, 
> the address of the object could have been taken to create a 
> pointer to the object (and that could have been done in code 
> for removed from the code that's using the lock with all of the 
> code around the lock being @safe). Heck, there could even be 
> references to data within the object rather than to the object 
> itself which are available elsewhere, meaning that part of the 
> object is protected by the lock and part isn't. If any 
> reference to any part of the data exists anywhere in the 
> program, then it's possible for another thread to access the 
> data at the same time that it's locked by the mechanism that 
> you've proposed.
Ok, that whole reference stuff is always a problem. Why not 
simply forbid it? You can't reference shared variables (outside 
locked blocks), you can only copy them. We can later relax that 
rule if some safe ways to allow that are found. I can't see why 
that should hinder us to make the more practical usecases safe 
for now.

>
> In order for the compiler to be able to actually guarantee that 
> it's safe to remove shared from an object, it has to be able to 
> guarantee that there are no other references to any part of 
> that object which exist in the program. That's why TDPL 
> synchronized classes are so locked down. Without that, other 
> references to the data could exist. And even with all of the 
> restrictions that they have, the compiler would still only be 
> able to remove the outer layer of shared - the layer directly 
> in the class - not any more than that. With something as 
> free-form as you're proposing, the compiler can't guarantee 
> anything, let alone that it's thread-safe to completely remove 
> shared from an object within that block.
>
> If D had ownership semantics baked into its type system, then 
> we could probably do more, but as it is, the compiler is _very_ 
> limited in its ability to know that no other references to any 
> portion of an object exist.
If we forbid them (for now), the compiler is well able to.

> Even scope is only able to do its job by restricting the 
> operations that are allowed on a scope object, not by actually 
> tracking an object's lifetime.
Yes, and that's fine. It isn't necessary that everything is 
possible with an object, but it should at least be useful for 
SOME task.

> For any proposal you might have with regards to how we might be 
> able to have the compiler safely remove shared from an object 
> for us, you're going to have to be able to prove that no other 
> references to any piece of that object could possibly exist or 
> that there's no way that any reference to any portion of that 
> object could be accessed without the same mutex being locked at 
> every point that it's accessed.
Understood.

> And it wouldn't surprise me if someone else were able to point 
> out why even that wasn't enough because of some detail I'm not 
> thinking of at the moment. Having the compiler be able to prove 
> that a piece of code is thread-safe such that shared can be 
> safely removed automatically from anything is incredibly 
> difficult.
shared shouldn't be removed from an object, but it can only be 
modyfied if it is locked. Removing shared (with a cast) is system 
stuff and should be out of scope for any safety related proposal 
(including mine), because with system stuff you can destroy any 
kind of safety.

If you don't remove shared, you can easily apply rules like 
forbid to take it's address or such. If you remove it, that makes 
it much harder (and isn't useful anyway).

I still think my proposal could work (provide provable 
thread-safety for shared objects) in a limited but useful way 
(only mutex, no references), and should be relatively easy to 
implement.
If you want more complex stuff, that's still possible in the same 
way it currently is: cast shared away together with all 
guarantees and verify manually that it works, just like in C++.