shared - i need it to be useful

Joakim dlang at joakim.fea.st
Thu Oct 18 02:02:58 UTC 2018


On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
> On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via 
> Digitalmars-d <digitalmars-d at puremagic.com> wrote:
>>
>> On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
>> > On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via 
>> > Digitalmars-d <digitalmars-d at puremagic.com> wrote:
>> >>
>> >> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
>> >>
>> >> > I've said this a bunch of times, there are 2 rules:
>> >> > 1. shared inhibits read and write access to members
>> >> > 2. `shared` methods must be threadsafe
>> >> >
>> >> >>From there, shared becomes interesting and useful.
>> >>
>> >> Oh God...
>> >>
>> >> void atomicInc(shared int* i) { /* ... */ }
>> >>
>> >> Now what? There are no "methods" for ints, only UFCS. Those 
>> >> functions can be as safe as you like, but if you allow 
>> >> implicit promotion of int* to shared int*, you *allow 
>> >> implicit races*.
>> >
>> > This function is effectively an intrinsic. It's unsafe by 
>> > definition.
>>
>> Only if implicit conversion is allowed. If it isn't, that's 
>> likely @trusted, and this:
>>
>> void atomicInc(ref shared int);
>>
>> can even be @safe.
>
> In this case, with respect to the context (a single int) 
> atomicInc()
> is ALWAYS safe, even with implicit conversion. You can 
> atomicInc() a
> thread-local int perfectly safely.
>
>> > It's a tool for implementing threadsafe machinery.
>> > No user can just start doing atomic operations on random ints
>> > and say
>> > "it's threadsafe", you must encapsulate the threadsafe
>> > functionality
>> > into some sort of object that aggregates all concerns and
>> > presents an
>> > intellectually sound api.
>>
>> Threadsafety starts and ends with the programmer. By your 
>> logic *all* functions operating on `shared` are unsafe then. 
>> As far as compiler is concerned, there would be no difference 
>> between these two:
>>
>> struct S {}
>> void atomicInc(ref shared S);
>>
>> and
>>
>> struct S { void atomicInc() shared { /* ... */ } }
>>
>> The signatures of those two functions are exactly the same. 
>> How is that different from a function taking a shared int 
>> pointer or reference?
>
> It's not, atomicInc() of an int is always safe with respect to 
> the int itself.
> You can call atomicInc() on an unshared int and it's perfectly 
> fine,
> but now you need to consider context, and that's a problem for 
> the
> design of the higher-level scope.
>
> To maintain thread-safety, the int in question must be 
> appropriately contained.
>
> The problem is that the same as the example I presented before, 
> which I'll repeat:
>
> struct InvalidProgram
> {
>   int x;
>   void fun() { ++x; }
>   void gun() shared { atomicInc(&x); }
> }
>
> The method gun() (and therefore the whole object) is NOT 
> threadsafe by
> my definition, because fun() violates the threadsafety of gun().
> The situation applies equally here that:
> int x;
> atomicInc(&x);
> ++x; // <- by my definition, this 'API' (increment an int) 
> violates
> the threadsafety of atomicInc(), and atomicInc() is therefore 
> not
> threadsafe.
>
> `int` doesn't present a threadsafe API, so int is by 
> definition, NOT threadsafe. atomicInc() should be @system, and 
> not @trusted.
>
> If you intend to share an int, use Atomic!int, because it has a 
> threadsafe API.
> atomicInc(shared int*) is effectively just an unsafe intrinsic, 
> and
> its only use is at ground-level implementation of threadsafe
> machinery, like malloc() and free().
>
>> > Let me try one:
>> >
>> > void free(void*) { ... }
>> >
>> > Now what? I might have dangling pointers... it's a 
>> > catastrophe!
>>
>> One could argue that it should be void free(ref void* p) { /* 
>> ...
>> */ p = null; }
>
> void *p2 = p;
> free(p);
> p2.crash();
>
>> As a matter of fact, in my own allocators memory blocks 
>> allocated by them are passed by value and are non-copyable, 
>> they're not just void[] as in std.experimental.allocator. One 
>> must 'move' them to pass ownership, and that includes 
>> deallocation. But that's another story altogether.
>
> Right, now you're talking about move semantics to implement 
> transfer of ownership... you might recall I was arguing this 
> exact case to express transferring ownership of objects between 
> threads earlier. This talk of blunt casts and "making sure 
> everything is good" is all just great, but it doesn't mean 
> anything interesting with respect to `shared`. It should be 
> interesting even without unsafe casts.
>
>> > It's essentially the same argument.
>> > This isn't a function that professes to do something that
>> > people might
>> > misunderstand and try to use in an unsafe way, it's a 
>> > low-level
>> > implementation device, which is used to build larger *useful*
>> > constructs.
>>
>> You're missing the point, again. You have an int. You pass a 
>> pointer to it to some API that takes an int*. You continue to 
>> use your int as just an int.
>
> You have written an invalid program. I can think of an infinite 
> number
> of ways to write an invalid program.
> In this case, don't have an `int`, instead, have an Atomic!int; 
> you
> now guarantee appropriate access, problem solved!
> If you do have an int, don't pass it to other threads at random 
> when
> you don't have any idea what they intend to do with it! That's 
> basic
> common sense. You don't pass a pointer to a function if you 
> don't know
> what it does with the pointer!
>
>> The API changes, and now the function
>> you called previously takes a shared int*. Implicit conversion
>> works, everything compiles, you have a race. Now, that's of
>> course an extremely stupid scenario.
>
> Yes.
>
>> The point is: the caller of
>> some API *must* assert that they indeed pass shared data. It's
>> insufficient for the API alone to "promise" taking shared data.
>> That's the difference with promotion to `const`.
>
> The caller doesn't care if it's true that the callee can't do 
> anything
> with it that's unsafe anyway.
> We effect that state by removing all non-threadsafe access.

Rather than answering all objections in this thread in detail, a 
mistake I've made before too, I suggest you put up a 3-5 page 
document somewhere explaining your proposal in detail. You can 
use the feedback here as a guide on what to explain more and what 
not to. Be sure to include code samples of how you see everything 
ultimately working in your external document.

Two mistakes you may be making in writing responses in this 
thread, which I've made before too:

1. Assuming people read anything more than fragments of what 
you're writing. My experience suggests people just read bits and 
pieces quickly, then compose them together in the way that makes 
most sense to them based on how _past_ technology works.

2. Assuming people have any idea how the underlying 
implementation of shared and multi-threading works. I know little 
to nothing about how shared systems work and how that interacts 
with type systems, which is why I haven't responded to your 
proposal, but that doesn't stop others from responding who may 
not know much more.

Neither of these will be solved by writing an external document, 
but at least Walter and others more seriously interested will 
benefit from a better-motivated explanation with more detail. I 
won't read it, ;) but I think we can avoid another long thread 
like this if you go this route. I'm planning the same approach 
with the technical topic I raised before, along with working code.


More information about the Digitalmars-d mailing list