Struggling to implement parallel foreach...

[Manu]

On Fri, Jun 14, 2019 at 9:10 PM Timon Gehr via Digitalmars-d
<digitalmars-d at puremagic.com> wrote:
>
> On 15.06.19 04:40, Manu wrote:
> > On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d
> > <digitalmars-d at puremagic.com> wrote:
> >>
> >> On 14.06.19 20:51, Manu wrote:
> >>> On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
> >>> <digitalmars-d at puremagic.com> wrote:
> >>>>
> >>>> On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
> >>>>> Right, exactly... the compile error should be at the assignment
> >>>>> of the
> >>>>> not-shared closure to the shared delegate. The error would be
> >>>>> something like "can't assign `Context*` to `shared(Context)*`".
> >>>>> So, the function can certainly exist, but for shared, you
> >>>>> should get
> >>>>> an error when you attempt to call it passing the closure to the
> >>>>> function.
> >>>>
> >>>> The shared function attribute means that its context is shared
> >>>> similar to object methods, the error is an attempt to implicitly
> >>>> leak unshared data into that shared context, that's why indeed
> >>>> function itself is incorrect. Declare the variable as shared and
> >>>> it will work.
> >>>
> >>> No, you've misunderstood. The qualifier does NOT apply to the context
> >>> as it should, that's the issue I'm reporting.
> >>
> >> There is a bug, but It appears you don't understand what it is. Why do
> >> you insist on reporting the bug in a particular wrong way instead of
> >> trying to understand what is actually going on?
> > ...
>
> (I'm going to ignore your other post, as it is just more rambling.)
>
> Note that this part of the language is extremely poorly implemented in
> DMD right now. There are quite a few independent bugs plaguing local
> function and delegate qualifiers. But the fact that you can't access an
> unshared variable from a `shared` local function is by design; it is not
> one of those bugs. The fact that `const` on local functions does not
> work properly has no bearing on `shared` on local functions. DMD
> implements those things independently.
>
> > So, what is actually going on?
>
> (NOTE: I am first explaining the actual design here, not the buggy
> implementation. This would make your parallel `foreach` code compile.
> You can't argue against what is below on the basis that it is obviously
> nonsense because it prevents your parallel `foreach` from compiling.)
>
> For member functions, the qualifiers affect the this pointer. I.e., you
> need to construct a differently-qualified receiver to call a `shared` or
> `immutable` member function. This is not the only behavior that would
> make sense. Qualifiers on member functions could also just restrict how
> that member function accesses other members.
>
> For local functions, the qualifiers specify how the context is
> _accessed_, not how the entire thing is qualified. If your local
> function is `shared` that means the function may only access `shared` data.
>
> void main(){
>      int x; // not shared, yet part of stack frame
>      shared(int) y;
>      int foo()shared{
>          // x=2; // error, x is not shared
>          return y; // ok
>      }
>      // ...
> }
>
> Similarly, if your local function is `immutable`, it may only access
> `immutable` data. If your local function is `const`, it may only access
> `const` data, but as everything implicitly converts to `const`, it can
> actually access everything, it may just not modify it, accessed
> variables are `const`-qualified. (It's a bug in DMD that they are not.)
>
> (Const is a special case because this is the only case where you can
> interpret what is going on as slapping the `const` qualifier on the
> entire context. It is not necessarily the best way to think about what
> is going on.)
>
> This is the way this was intended to work, but apparently it was never
> fully implemented, leaving behind quite a few type system holes, for
> example:
>
> void main(){
>      int x;
>      void bar(){
>          x=2;
>      }
>      void foo()shared{
>          bar(); // this shouldn't compile, but it does
>      }
> }
>
> Local functions should _infer_ those attributes. So if your function
> (like the implicitly-generated lambda representing your `foreach` body),
> only accesses `shared` variables, it should be automatically
> `shared`-qualified.
>
> > (I believe I'm aware what is going on, because behaviour is self-evident).
>
> (You have demonstrated that this is not the case.)

No, I genuinely understand. You've demonstrated that I know exactly
what was intended, and it's terrible.
With this semantic, it's true I can call the function, but the
function fundamentally doesn't work. There's no point making the
function callable if the function can't do anything. I need a useful
function to call before worrying about how to call it. We can get
there in better ways without a complex web of special-case rules.
This existing semantic expects that the context has shared objects
floating around to interact with. When have you ever declared a shared
object anywhere?

> > And more importantly, how is it useful?
>
> You have argued that there shouldn't be any way to call
> `shared`-qualified local functions. I don't understand how you can
> possibly think that behavior is useful. It would preclude your parallel
> `foreach` code from working!

I can't move on that front without fixing this first.
I have a clear plan to get there, but this design choice is blocking
progress. It seems pointless, can you show me any cases demonstrating
where this case-specific complexity is useful?

> The intended design is useful in the sense that it would make your
> parallel `foreach` code work in a compiler-checked thread safe way,
> because the compiler would simply automatically check that the loop body
> only accesses shared variables (or variables local to the loop body, of
> course).
>
> > Why is it so useful that it should it violate default behaviour, and expectation?
>
> It does not.

It does though, there is a completely different set of semantics
applied to a local function than to normal methods. That's extremely
surprising.
The weird thing is, both methods AND local functions can be assigned
to delegates... delegates do not distinguish between these 2 distinct
sets of semantics, and I believe that's why these
pipe-through-a-delegate-to-do-invalid-behaviour bugs exist. That would
require even more special-case code to correct.
A better design would be that no special case exists, and then no
special case corrections to special cases should exist either.
This is a bad design. Local functions context pointers should behave
identical to methods, and then delegates, and code that calls
delegates know what semantics they're dealing with.
A 'this' pointer is a 'this' pointer, don't make it more complicated than that.

> > I know how it's not useful.
> >
>
> I am not defending the fact that your parallel `foreach` code does not
> compile! This has to be fixed.
>
> But if it is fixed, and if D passes a shared delegate to your parallel
> `foreach`, where `shared` means it is @safe to call that delegate from
> other threads,

I don't believe that guarantee is possible. `shared` is fundamentally
unsafe, we've argued this for a long time.
It is possible to build safe tools with shared, but at the low level, it is not.

> how would that not be useful?

Because the function can't access anything.
This is the essential case:

struct S
{
  void method() shared;
}

void test()
{
  S s;
  void localFun() shared
  {
    s.method(); // <- s must transitively receive const from the
context pointer, otherwise this doesn't work
  }
}

Current semantics would reject this, because s is not shared, and
therefore not accessible by localFun.
The unnecessary complexity inhibits the only useful thing that a
shared function can do.