Struggling to implement parallel foreach...

[Manu]

On Tue, Jun 18, 2019 at 8:10 AM Timon Gehr via Digitalmars-d
<digitalmars-d at puremagic.com> wrote:
>
> On 17.06.19 08:15, Manu wrote:
> >     Closures with an immutable-qualified context can only
> >     capture immutable variables. What is so surprising about this?
> >
> >
> > What's the use of that? Can you point me at any code like that?
>
> No, I don't have in my head an index of all existing D code keyed on
> which features it is using (not even my own), but I know how to design a
> programming language correctly. One reason why I don't have a lot of
> examples demonstrating usage of qualifiers is that I avoid them as well
> as I can -- their implementation is broken, and currently I have no
> interest in dealing with all of those bugs.
>
> If I spent an unreasonable amount of time I would probably be able to
> produce plausible examples, but this would be a waste of my time,
> because it is not what I am good at or enjoy doing.
>
> It would be a lot more productive if you produced the examples and
> trusted me more when I explain the language design considerations. I
> also want the language to improve, this is why I push back against bad
> aspects of otherwise good ideas. It does not mean I oppose the general
> direction things are moving.
>
> > Make a const local function if you want to stop it mutating stuff...
>
> That doesn't stop others from mutating. The _only way_ to get a strongly
> pure delegate literal is to qualify its context immutable. I hope this
> is reason enough without a concrete usage example.
>
> > what's interesting about an immutable local function?
> > ...
>
> You can call an immutable-qualified delegate only if the context pointer
> is immutable-qualified. If it is const-qualified, you can't do that
> without breaking the type system.
>
> It would be so weird if the only way to make a strongly pure delegate
> would be to create a local `static struct`, explicitly capture the
> `immutable` variables in it's members, and take the address of an
> `immutable` member function of the resulting immutable `struct` instance.
>
> Maybe this way of thinking about it helps: You don't capture the entire
> stack frame and the context does not point to the entire stack frame. It
> only points to the variables that were actually captured.
>
>
> >      > inout is fine too, just like const.
> >
> >     Absolutely not. You can't implicitly promote stuff to `inout`.
> >
> >
> > What are you taking about? inout accepts mutable, const, or immutable. I
> > don't know what you could mean?
>
> What I meant is `inout` does not work "just like const". Maybe this is
> not what you meant to imply. Conceptually, `inout` is a polymorphic
> parameter. It represents an actual qualifier at each specific point in
> the program, you just don't know what it is. You can't implicitly
> convert a mutable variable into an `inout` variable, because it is
> possible that `inout` means `immutable` during the actual execution of
> the program.
>
> > Context objects are mutable, unless called via another local function
> > where they would have gained the respective qualifier, and that would
> > interact with inout correctly, eliminating some existing
> > false-mutability bugs.
> > ...
>
> I hate how complicated it is to explain why this is nonsense, especially
> because it is only a tangent in your post, but I'll bite. `inout`
> strikes again. (BTW: How much time do you spend thinking about your
> posts in this thread? I am wasting hours upon hours. I can't keep this
> up much longer, I have other things I need to do... Please start
> figuring out the holes in your suggestions on your own. Please be more
> careful with the assertions that you make.)
>
> The issue with `inout` is that there is scope confusion: there can be
> multiple enclosing function scopes, all of which have an `inout`
> qualifier, and all of those different polymorphic parameters have the
> same name: "inout". Right now the language deals with this (somewhat
> unsuccessfully implemented in DMD!), by effectively disallowing nested
> `inout` functions (their `inout` is treated as being the same as that of
> the enclosing function).
>
> Therefore, right now there is no way to "interact with inout correctly"
> in the way you suggest, the `inout` qualifier on a nested function
> cannot be instantiated with a concrete value, because this happens only
> once the _outermost_ function gets called.
>
> Of course, you will now just say that this is all bollocks and we should
> change all of that too.
>
> So let's summarize. What you are saying is:
> 1. Any local function can capture any local variable from an enclosing
> scope.
>
> 2. The type of those variables pick up the context qualifier of the
> local functions trough which they are captured.
>
> 3. `inout` local functions can be called from other local functions and
> their context `inout` is instantiated with the qualifier of the other
> local function.
>
> But you can't actually qualify a `foo` context with `bar`s `inout`,
> because you would immediately get `inout` confusion. Of course, the
> compiler would still happily do it. So your proposal would lead to at
> least one `inout` bug:
>
> int* foo(inout(int)* x)@safe{
>      inout(int)* bar()inout{
>          return x; // ok, type of x is inout(inout(int)*)
>      }
>      int* baz(){ return bar(); }
>      return baz();
> }
>
> void main()@safe{
>      immutable(int)* x=...;
>      int* y=foo(x);
>      // x and y alias, UB
>      assert(x is y);
> }
>
> (The function `baz` is just to make it very explicit that this is what
> you propose, I think you agree that `foo` could return bar(dummy)
> directly with the same result.)
>
> The right solution would actually be to have proper scoping for `inout`,
> where additionally you can have multiple names for `inout`, e.g.
> `inout!"foo"` and `inout!"bar"`. This would fix every `inout`
> false-mutability bug, with your weird context interpretation or without.
> But then the correct design is that a function with `inout` context
> would only be able to capture variables with the right type of `inout`,
> the one as which its context is qualified. Otherwise you couldn't call
> `inout` local functions.
>
>
> >      > shared is the interesting one; we shouldn't be able to pass the
> >      > capture to the function because Context* -> shared(Context)*, but we
> >      > can start to talk about ways this can work.
> >      > One way is that the promotion is actally perfectly valid! Because you
> >      > are calling a local function from the local thread; so the shared
> >      > function will have a shared reference to the local context only for
> >      > the life of the function call, and when it returns, the shared
> >      > reference must end with the function. We can do this by declaring the
> >      > local function: `void localFun(scope shared(Context)* ctx);`
> >      > A shared local function is valid so long as that function does NOT
> >      > escape any of those shared references. It might want to attribute
> >      > `return` too.
> >
> >     This idea is not at all contingent on the nonsensical
> >
> >
> > How is uniformity nonsense?
>
> Probably I should have been more careful in my first post. I did
> explicitly say that the local function meaning could be useful for
> member functions and not the other way around, but I guess you ignored
> that and just concluded "non-uniform".
>
> It's not really all that non-uniform, I can implement `immutable`
> capturing manually like this:
>
> // version with closures:
> int delegate()immutable bar(){
>      immutable(int) x = 3;
>      int foo()immutable{
>          return x;
>      }
>      return &foo;
> }
> // version without closures:
> int delegate()immutable bar(){
>      struct StackFrame{
>          immutable(int) x = 3;
>      }
>      auto theFrame=new StackFrame();
>      struct Closure{
>          int* xp;
>          int foo()immutable{
>              return *xp;
>          }
>      }
>      return &new immutable(Closure)(&theFrame.x).foo;
> }
>
> Of course, the compiler will do with one heap allocation to get an
> equivalent result, which is currently impossible to do with member
> functions.
>
> > Are you saying that qualified methods are nonsense?
>
> Qualifying the `this` pointer is not nonsense, because you can actually
> get a value of that type. Qualifying the full stack frame is nonsense,
> because there is no way to create an instance of a qualified stack
> frame, except for qualifiers where you can implicitly promote an
> unqualified stack frame, and there is a very useful way to interpret
> qualified capturing, which is easy to understand (see above).

Right... this is my entire point, and why I say that immutable local
methods make no sense.
Local methods are not yet delegates; they may be captured into a
delegate, but that's an interesting operation, and that's also the
moment where `immutable` becomes interesting.
At the time you want to capture a delegate from a local function, you can copy


If we create a closure by copying referenced data into an immutable
object, then it doesn't depend on referenced elements being immutable,
only copied to immutable.
Such a closure could be allocated on the stack for scope delegates.

> However, I do indeed dislike that there is no way to say a member
> function will only access `immutable` members:
>
> struct S{
>      immutable x;
>      int y;
>      int foo(){ // only accesses immutable data
>          return x;
>      }
> }
>
> void main(){
>       // could work, but typesystem can't show it:
>      int delegate()immutable dg=&new S(1,2).foo;
> }
>
> There is a workaround:
>
> struct T{
>      int x;
>      int foo()immutable{
>          return x;
>      }
> }
> struct S{
>      immutable(T) t;
>      int y;
>      this(int x,int y){ t=T(x); y=y; }
> }
>
> void main(){
>      int delegate()immutable dg=&new S(1,2).t.foo;
> }
>
> This workaround would also work in a version of the language that
> implements your set of rules (with `inout` bugs and all):
>
> But I'd much rather write this than the above:
>
> void main(){
>      immutable(int) x=1;
>      int y=2;
>      int delegate()immutable dg=()=>x;
>      writeln(dg());
> }
>
> With your suggested changes, this would not compile, for no benefit at
> all. It would stop qualifier inference for nested functions in its tracks.
>
> > How could applying proven, uniform, and predictable rules be
> > considered nonsense?
>
> The nonsense is not in how you apply the rules (that part is fine, you
> correctly derive the consequences), it is in how you select the rules
> and how you justify selecting those rules.
>
> > Can you show me how these special-case rules are useful?
>
> I have argued above that there is no special case.
>
> > Qualified local functions are extremely rare;
>
> Because they don't work properly, which you have already noticed. E.g.,
> above I should be allowed to return `int delegate()` instead of `int
> delegate()immutable`, but the compiler won't let me. It's full of bugs.
> I would use qualifiers if they worked, and I would use them more if I
> had to write multithreaded code.
>
> > I suspect someone just dun some design
> > that sounded cool and felt chuffed, but probably never used it.
> >
> >     parts of your
> >     vision, but I don't know if there is a simple way to make this work. It
> >     will require more care.
> >
> >
> > I agree there's certainly no simple way to do it assuming the current
> > design;
>
> I have shown you how to manually construct `immutable` closures with
> nothing but structs and member functions. I hope that clarifies that
> this assertions is plain nonsense.
>
> > I did try and imagine a solution extensively under the existing
> > implementation for quite some time, but I don't believe it's possible
> > without effectively transforming it towards the default semantics.
> > ...which isn't actually surprising! You shouldn't be surprised that
> > uniform semantics interact properly with all the other existing
> > language. If the design just worked like any normal qualified this
> > pointer, there'd be nothing to think about other than how to call the
> > shared function, which I think is a problem we can defeat. Everything
> > else would work elegantly with no special code in the compiler.
> > ...
>
> There would be the same amount of special code in the compiler, because
> stack frames are not structs.
>
> >      >>From there, parallel foreach can be implemented as a @trusted
> >     function.
> >      >
> >
> >     So your evil master plan is:
> >
> >     1. Make qualified local functions useless.
> >
> >
> > Not at all, const and inout work as expected
>
> I agree: `const` works fine, `inout` continues to be buggy, as expected.
>
> > (surely the 99% case),
>
> I hate 99% features.
>
> > and reflective meta will actually work too.
>
> When does it not work? (I am not doubting that it does not work, but I
> very much doubt it is because qualified closures are designed usefully.)
>
> > Existing bugs are all eliminated instantly.
>
> That's plain snake oil and completely untrue. Have you ever seen the DMD
> source code?
>
> > `immutable` has no useful meaning anyway
>
> Wtf.
>
> > (just use const? The
> > thing about local functions is that they're local. It's not API
> > material),
>
> Yes, it is. Qualified delegate types can appear in APIs. Your very first
> example in this thread is an example of how it is API material.
>
> > shared is the only interesting case that's affected, and I
> > can't imagine another way to make it useful.
> > ...
>
> I hope after reading this and the previous post you will notice that
> there is actually no blocker.
>
> >     2. Make qualified local functions more useful.
> >
> >
> > If you can do this without 1, then great.
>
> I can.
>
> > I couldn't imagine a solution.
> > The problem is specifically the non-uniformity and special case rules.
> > ...
>
> Do you still feel that way now?
>
> >     3. Profit.
> >
> >     Just skip phase 1, please.
> >
> >
> > I've asked a couple of times for a demonstration of how existing
> > semantics are useful? It's certainly complex, non-uniform, and surprising.
> > The closure has un-qualified objects in it despite the function
> > qualification, which means typeof() doesn't work right, and derivative
> > meta that depends on that is broken by consequence.
> > ...
>
> The closure does not have unqualified objects in it. Why is it a problem
> that it is not necessarily contiguous in memory because the compiler
> optimizes away allocations?
>
> > Explain to me how qualifying the context makes it useless? How does it
> > make anything that works right now stop working?
>
> I have shown a few examples in this post and argued why they are useful.
> (E.g., strongly pure delegates.)
>
> > I think that makes it uniform, only meaningfully affects shared (in a
> > positive way), and removes a mountain of complex special case code.
> > ...
>
> Step 1 only removes functionality, it does not eliminate any blockers
> for anything, including your proposed redefinition of `shared` to mean
> `threadsafe`.
>
> > But you know what, do what you want, I'll shut up... If it just gets
> > fixed, I won't complain.
>
> Having wasted a lot of time on this thread, it would still be great if
> you made an effort to understand my points.

I started replying here; but in a large part, you're telling me things
I am absolutely aware of as if they're details I missed... and where I
disagree, you'll just hate anyway.
I genuinely don't intend to waste your time.

I believe 80% of the issue you have with what I'm saying is because
you either misunderstand what I say, I poorly communicate what I mean,
or because I skip the broader details of my suggestions because it's
too long.
I'm starting to see that it *might* even be possible that our visions
are actually very closely aligned, but presented slightly differently.

I can't work via email, it kills me, and it wastes our time. Are you
in the US? Where do you live? I will fly to you, we can fix it with a
whiteboard because forums are the most bullshit form of communication
ever divised by man.

This is what dconf should be is for, but instead we spend the whole
time watching lectures.

I wish I had the time + energy to fork the language and just prove
shit, rather than waste time talking about it.