Struggling to implement parallel foreach...

[Timon Gehr]

On 17.06.19 08:15, Manu wrote:
>     Closures with an immutable-qualified context can only
>     capture immutable variables. What is so surprising about this?
> 
> 
> What's the use of that? Can you point me at any code like that?

No, I don't have in my head an index of all existing D code keyed on 
which features it is using (not even my own), but I know how to design a 
programming language correctly. One reason why I don't have a lot of 
examples demonstrating usage of qualifiers is that I avoid them as well 
as I can -- their implementation is broken, and currently I have no 
interest in dealing with all of those bugs.

If I spent an unreasonable amount of time I would probably be able to 
produce plausible examples, but this would be a waste of my time, 
because it is not what I am good at or enjoy doing.

It would be a lot more productive if you produced the examples and 
trusted me more when I explain the language design considerations. I 
also want the language to improve, this is why I push back against bad 
aspects of otherwise good ideas. It does not mean I oppose the general 
direction things are moving.

> Make a const local function if you want to stop it mutating stuff...

That doesn't stop others from mutating. The _only way_ to get a strongly 
pure delegate literal is to qualify its context immutable. I hope this 
is reason enough without a concrete usage example.

> what's interesting about an immutable local function?
> ...

You can call an immutable-qualified delegate only if the context pointer 
is immutable-qualified. If it is const-qualified, you can't do that 
without breaking the type system.

It would be so weird if the only way to make a strongly pure delegate 
would be to create a local `static struct`, explicitly capture the 
`immutable` variables in it's members, and take the address of an 
`immutable` member function of the resulting immutable `struct` instance.

Maybe this way of thinking about it helps: You don't capture the entire 
stack frame and the context does not point to the entire stack frame. It 
only points to the variables that were actually captured.


>      > inout is fine too, just like const.
> 
>     Absolutely not. You can't implicitly promote stuff to `inout`.
> 
> 
> What are you taking about? inout accepts mutable, const, or immutable. I 
> don't know what you could mean?

What I meant is `inout` does not work "just like const". Maybe this is 
not what you meant to imply. Conceptually, `inout` is a polymorphic 
parameter. It represents an actual qualifier at each specific point in 
the program, you just don't know what it is. You can't implicitly 
convert a mutable variable into an `inout` variable, because it is 
possible that `inout` means `immutable` during the actual execution of 
the program.

> Context objects are mutable, unless called via another local function 
> where they would have gained the respective qualifier, and that would 
> interact with inout correctly, eliminating some existing 
> false-mutability bugs.
> ...

I hate how complicated it is to explain why this is nonsense, especially 
because it is only a tangent in your post, but I'll bite. `inout` 
strikes again. (BTW: How much time do you spend thinking about your 
posts in this thread? I am wasting hours upon hours. I can't keep this 
up much longer, I have other things I need to do... Please start 
figuring out the holes in your suggestions on your own. Please be more 
careful with the assertions that you make.)

The issue with `inout` is that there is scope confusion: there can be 
multiple enclosing function scopes, all of which have an `inout` 
qualifier, and all of those different polymorphic parameters have the 
same name: "inout". Right now the language deals with this (somewhat 
unsuccessfully implemented in DMD!), by effectively disallowing nested 
`inout` functions (their `inout` is treated as being the same as that of 
the enclosing function).

Therefore, right now there is no way to "interact with inout correctly" 
in the way you suggest, the `inout` qualifier on a nested function 
cannot be instantiated with a concrete value, because this happens only 
once the _outermost_ function gets called.

Of course, you will now just say that this is all bollocks and we should 
change all of that too.

So let's summarize. What you are saying is:
1. Any local function can capture any local variable from an enclosing 
scope.

2. The type of those variables pick up the context qualifier of the 
local functions trough which they are captured.

3. `inout` local functions can be called from other local functions and 
their context `inout` is instantiated with the qualifier of the other 
local function.

But you can't actually qualify a `foo` context with `bar`s `inout`, 
because you would immediately get `inout` confusion. Of course, the 
compiler would still happily do it. So your proposal would lead to at 
least one `inout` bug:

int* foo(inout(int)* x)@safe{
     inout(int)* bar()inout{
         return x; // ok, type of x is inout(inout(int)*)
     }
     int* baz(){ return bar(); }
     return baz();
}

void main()@safe{
     immutable(int)* x=...;
     int* y=foo(x);
     // x and y alias, UB
     assert(x is y);
}

(The function `baz` is just to make it very explicit that this is what 
you propose, I think you agree that `foo` could return bar(dummy) 
directly with the same result.)

The right solution would actually be to have proper scoping for `inout`, 
where additionally you can have multiple names for `inout`, e.g. 
`inout!"foo"` and `inout!"bar"`. This would fix every `inout` 
false-mutability bug, with your weird context interpretation or without. 
But then the correct design is that a function with `inout` context 
would only be able to capture variables with the right type of `inout`, 
the one as which its context is qualified. Otherwise you couldn't call 
`inout` local functions.


>      > shared is the interesting one; we shouldn't be able to pass the
>      > capture to the function because Context* -> shared(Context)*, but we
>      > can start to talk about ways this can work.
>      > One way is that the promotion is actally perfectly valid! Because you
>      > are calling a local function from the local thread; so the shared
>      > function will have a shared reference to the local context only for
>      > the life of the function call, and when it returns, the shared
>      > reference must end with the function. We can do this by declaring the
>      > local function: `void localFun(scope shared(Context)* ctx);`
>      > A shared local function is valid so long as that function does NOT
>      > escape any of those shared references. It might want to attribute
>      > `return` too.
> 
>     This idea is not at all contingent on the nonsensical
> 
> 
> How is uniformity nonsense?

Probably I should have been more careful in my first post. I did 
explicitly say that the local function meaning could be useful for 
member functions and not the other way around, but I guess you ignored 
that and just concluded "non-uniform".

It's not really all that non-uniform, I can implement `immutable` 
capturing manually like this:

// version with closures:
int delegate()immutable bar(){
     immutable(int) x = 3;
     int foo()immutable{
         return x;
     }
     return &foo;
}
// version without closures:
int delegate()immutable bar(){
     struct StackFrame{
         immutable(int) x = 3;
     }
     auto theFrame=new StackFrame();
     struct Closure{
         int* xp;
         int foo()immutable{
             return *xp;
         }
     }
     return &new immutable(Closure)(&theFrame.x).foo;
}

Of course, the compiler will do with one heap allocation to get an 
equivalent result, which is currently impossible to do with member 
functions.

> Are you saying that qualified methods are nonsense?

Qualifying the `this` pointer is not nonsense, because you can actually 
get a value of that type. Qualifying the full stack frame is nonsense, 
because there is no way to create an instance of a qualified stack 
frame, except for qualifiers where you can implicitly promote an 
unqualified stack frame, and there is a very useful way to interpret 
qualified capturing, which is easy to understand (see above).

However, I do indeed dislike that there is no way to say a member 
function will only access `immutable` members:

struct S{
     immutable x;
     int y;
     int foo(){ // only accesses immutable data
         return x;
     }
}

void main(){
      // could work, but typesystem can't show it:
     int delegate()immutable dg=&new S(1,2).foo;
}

There is a workaround:

struct T{
     int x;
     int foo()immutable{
         return x;
     }
}
struct S{
     immutable(T) t;
     int y;
     this(int x,int y){ t=T(x); y=y; }
}

void main(){
     int delegate()immutable dg=&new S(1,2).t.foo;
}

This workaround would also work in a version of the language that 
implements your set of rules (with `inout` bugs and all):

But I'd much rather write this than the above:

void main(){
     immutable(int) x=1;
     int y=2;
     int delegate()immutable dg=()=>x;
     writeln(dg());
}

With your suggested changes, this would not compile, for no benefit at 
all. It would stop qualifier inference for nested functions in its tracks.

> How could applying proven, uniform, and predictable rules be 
> considered nonsense?

The nonsense is not in how you apply the rules (that part is fine, you 
correctly derive the consequences), it is in how you select the rules 
and how you justify selecting those rules.

> Can you show me how these special-case rules are useful?

I have argued above that there is no special case.

> Qualified local functions are extremely rare;

Because they don't work properly, which you have already noticed. E.g., 
above I should be allowed to return `int delegate()` instead of `int 
delegate()immutable`, but the compiler won't let me. It's full of bugs. 
I would use qualifiers if they worked, and I would use them more if I 
had to write multithreaded code.

> I suspect someone just dun some design 
> that sounded cool and felt chuffed, but probably never used it.
> 
>     parts of your
>     vision, but I don't know if there is a simple way to make this work. It
>     will require more care.
> 
> 
> I agree there's certainly no simple way to do it assuming the current 
> design; 

I have shown you how to manually construct `immutable` closures with 
nothing but structs and member functions. I hope that clarifies that 
this assertions is plain nonsense.

> I did try and imagine a solution extensively under the existing 
> implementation for quite some time, but I don't believe it's possible 
> without effectively transforming it towards the default semantics.
> ...which isn't actually surprising! You shouldn't be surprised that 
> uniform semantics interact properly with all the other existing 
> language. If the design just worked like any normal qualified this 
> pointer, there'd be nothing to think about other than how to call the 
> shared function, which I think is a problem we can defeat. Everything 
> else would work elegantly with no special code in the compiler.
> ...

There would be the same amount of special code in the compiler, because 
stack frames are not structs.

>      >>From there, parallel foreach can be implemented as a @trusted
>     function.
>      >
> 
>     So your evil master plan is:
> 
>     1. Make qualified local functions useless.
> 
> 
> Not at all, const and inout work as expected

I agree: `const` works fine, `inout` continues to be buggy, as expected.

> (surely the 99% case),

I hate 99% features.

> and reflective meta will actually work too.

When does it not work? (I am not doubting that it does not work, but I 
very much doubt it is because qualified closures are designed usefully.)

> Existing bugs are all eliminated instantly.

That's plain snake oil and completely untrue. Have you ever seen the DMD 
source code?

> `immutable` has no useful meaning anyway

Wtf.

> (just use const? The 
> thing about local functions is that they're local. It's not API 
> material),

Yes, it is. Qualified delegate types can appear in APIs. Your very first 
example in this thread is an example of how it is API material.

> shared is the only interesting case that's affected, and I 
> can't imagine another way to make it useful.
> ...

I hope after reading this and the previous post you will notice that 
there is actually no blocker.

>     2. Make qualified local functions more useful.
> 
> 
> If you can do this without 1, then great.

I can.

> I couldn't imagine a solution. 
> The problem is specifically the non-uniformity and special case rules.
> ...

Do you still feel that way now?

>     3. Profit.
> 
>     Just skip phase 1, please.
> 
> 
> I've asked a couple of times for a demonstration of how existing 
> semantics are useful? It's certainly complex, non-uniform, and surprising.
> The closure has un-qualified objects in it despite the function 
> qualification, which means typeof() doesn't work right, and derivative 
> meta that depends on that is broken by consequence.
> ...

The closure does not have unqualified objects in it. Why is it a problem 
that it is not necessarily contiguous in memory because the compiler 
optimizes away allocations?

> Explain to me how qualifying the context makes it useless? How does it 
> make anything that works right now stop working?

I have shown a few examples in this post and argued why they are useful. 
(E.g., strongly pure delegates.)

> I think that makes it uniform, only meaningfully affects shared (in a 
> positive way), and removes a mountain of complex special case code.
> ...

Step 1 only removes functionality, it does not eliminate any blockers 
for anything, including your proposed redefinition of `shared` to mean 
`threadsafe`.

> But you know what, do what you want, I'll shut up... If it just gets 
> fixed, I won't complain.

Having wasted a lot of time on this thread, it would still be great if 
you made an effort to understand my points.