Struggling to implement parallel foreach...

[Timon Gehr]

On 15.06.19 04:40, Manu wrote:
> On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d
> <digitalmars-d at puremagic.com> wrote:
>>
>> On 14.06.19 20:51, Manu wrote:
>>> On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
>>> <digitalmars-d at puremagic.com> wrote:
>>>>
>>>> On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
>>>>> Right, exactly... the compile error should be at the assignment
>>>>> of the
>>>>> not-shared closure to the shared delegate. The error would be
>>>>> something like "can't assign `Context*` to `shared(Context)*`".
>>>>> So, the function can certainly exist, but for shared, you
>>>>> should get
>>>>> an error when you attempt to call it passing the closure to the
>>>>> function.
>>>>
>>>> The shared function attribute means that its context is shared
>>>> similar to object methods, the error is an attempt to implicitly
>>>> leak unshared data into that shared context, that's why indeed
>>>> function itself is incorrect. Declare the variable as shared and
>>>> it will work.
>>>
>>> No, you've misunderstood. The qualifier does NOT apply to the context
>>> as it should, that's the issue I'm reporting.
>>
>> There is a bug, but It appears you don't understand what it is. Why do
>> you insist on reporting the bug in a particular wrong way instead of
>> trying to understand what is actually going on?
> ...

(I'm going to ignore your other post, as it is just more rambling.)

Note that this part of the language is extremely poorly implemented in 
DMD right now. There are quite a few independent bugs plaguing local 
function and delegate qualifiers. But the fact that you can't access an 
unshared variable from a `shared` local function is by design; it is not 
one of those bugs. The fact that `const` on local functions does not 
work properly has no bearing on `shared` on local functions. DMD 
implements those things independently.

> So, what is actually going on?

(NOTE: I am first explaining the actual design here, not the buggy 
implementation. This would make your parallel `foreach` code compile. 
You can't argue against what is below on the basis that it is obviously 
nonsense because it prevents your parallel `foreach` from compiling.)

For member functions, the qualifiers affect the this pointer. I.e., you 
need to construct a differently-qualified receiver to call a `shared` or 
`immutable` member function. This is not the only behavior that would 
make sense. Qualifiers on member functions could also just restrict how 
that member function accesses other members.

For local functions, the qualifiers specify how the context is 
_accessed_, not how the entire thing is qualified. If your local 
function is `shared` that means the function may only access `shared` data.

void main(){
     int x; // not shared, yet part of stack frame
     shared(int) y;
     int foo()shared{
         // x=2; // error, x is not shared
         return y; // ok
     }
     // ...
}

Similarly, if your local function is `immutable`, it may only access 
`immutable` data. If your local function is `const`, it may only access 
`const` data, but as everything implicitly converts to `const`, it can 
actually access everything, it may just not modify it, accessed 
variables are `const`-qualified. (It's a bug in DMD that they are not.)

(Const is a special case because this is the only case where you can 
interpret what is going on as slapping the `const` qualifier on the 
entire context. It is not necessarily the best way to think about what 
is going on.)

This is the way this was intended to work, but apparently it was never 
fully implemented, leaving behind quite a few type system holes, for 
example:

void main(){
     int x;
     void bar(){
         x=2;
     }
     void foo()shared{
         bar(); // this shouldn't compile, but it does
     }
}

Local functions should _infer_ those attributes. So if your function 
(like the implicitly-generated lambda representing your `foreach` body), 
only accesses `shared` variables, it should be automatically 
`shared`-qualified.

> (I believe I'm aware what is going on, because behaviour is self-evident).

(You have demonstrated that this is not the case.)

> And more importantly, how is it useful? 

You have argued that there shouldn't be any way to call 
`shared`-qualified local functions. I don't understand how you can 
possibly think that behavior is useful. It would preclude your parallel 
`foreach` code from working!

The intended design is useful in the sense that it would make your 
parallel `foreach` code work in a compiler-checked thread safe way, 
because the compiler would simply automatically check that the loop body 
only accesses shared variables (or variables local to the loop body, of 
course).

> Why is it so useful that it should it violate default behaviour, and expectation?

It does not.

> I know how it's not useful.
> 

I am not defending the fact that your parallel `foreach` code does not 
compile! This has to be fixed.

But if it is fixed, and if D passes a shared delegate to your parallel 
`foreach`, where `shared` means it is @safe to call that delegate from 
other threads, how would that not be useful?