Discussion Thread: DIP 1041--Attributes for Higher-Order Functions--Community Review Round 1

[Q. Schroll]

On Monday, 12 April 2021 at 17:21:47 UTC, Timon Gehr wrote:
> On 12.04.21 16:44, Q. Schroll wrote:
>>
>> On Monday, 12 April 2021 at 11:05:14 UTC, Timon Gehr wrote:
>>> Unfortunately, it is not written too well: The reader gets 
>>> flooded with details way before being told what the problem 
>>> actually is or how the proposal addresses it.
>>
>>
>> Doesn't the Abstract explain what the problem is and give a 
>> general idea how it is addressed?
>
> It does not. It's generic fluff. It's only marginally more 
> explicit than: "there are problems and to address them we 
> should change the language".

I had a more detailed Abstract in previous drafts, but if you 
think I watered it down too much, I can add more details.

>>> As far as I can tell, this is trying to introduce attribute 
>>> polymorphism without actually adding polymorphism, much like 
>>> `inout` attempted and ultimately failed to do. I am very 
>>> skeptical. It's taking a simple problem with a simple 
>>> solution and addressing it using an overengineered 
>>> non-orthogonal mess in the hopes of not having to add 
>>> additional syntax.
>>
>> You're mistaken. You can take a look at the Alternatives for 
>> seemingly simple solutions. There ain't any.
>
> I know there are, and I literally state how to do it in the 
> quoted excerpt.

If by "quoted excerpt" you mean "As far as I can tell, this", I 
read it, but to be honest, I didn't really understand what 
attribute polymorphism really means. Googling "polymorphism" the 
closet I come to would be that a `@safe` delegate can be used in 
place of a `@system` delegate. This is already the case, I can't 
see how anything would "introduce" it.

> > You always have the problem of assigning the parameter in the 
> > functional unless it's `const` or another flavor of 
> > non-mutable.
>
> Assignments are not the problem, it's the inconsistent 
> interpretation of types using incompatible, special-cased 
> meanings.

Maybe I'm not creative enough for a proper solution, but I should 
be, since the problem is "easy".

>> If you don't go the `const` route, you have to deal with 
>> assignments to the parameter before it's called. You have to 
>> disallow assignments that, looking at the types, are a 1-to-1 
>> assignment. IMO, going via `const` is far more intuitive.
>
> It's a bad, non-orthogonal solution building on a compiler bug.
>
>> In fact, "not having to add additional syntax" was never the 
>> motivation for the proposal. Not having to introduce 
>> attributes _specific_ to higher-order function was.
>
> It adds higher-order specific rules to existing attributes 
> without a good reason, which is a lot worse.

I guess removing higher-order functions as a road-bump when it 
comes to attributes is a good reason. It's adding higher-order 
specific rules vs. adding another higher-order specific something.

>>> To add insult to injury, the first example that's shown in 
>>> the DIP as motivation abuses an existing type system hole.
>>
>> I disagree that it is a hole in the type system.
>
> You are wrong, and I am not sure how to make that point to you. 
> (When I tried last time, you just claimed that some other 
> well-documented intentionally-designed feature, like attribute 
> transitivity, is actually a bug.)
>
>> When having `qual₁(R delegate(Ps) qual₂)` where `qual₁` and 
>> `qual₂` are type qualifiers (`const`, `immutable`, etc.) it is 
>> practically most useful if `qual₁` only applies to the 
>> function pointer and (the outermost layer of) the context 
>> pointer while `qual₂` refers to the property of the context 
>> itself.
>
> That allows building a gadget to completely bypass transitivity 
> of qualifiers, including `immutable` and `shared`.

I had a look at [issue 
1983](https://issues.dlang.org/show_bug.cgi?id=1983) again where 
(I guess) the source of disagreement is how delegates should be 
viewed theoretically. If I understand you correctly, you say 
delegates *cannot possibly* be defined differently than having 
their contexts be literally part of them. I tried to explore 
definitions in which the context is *associated with* but not 
*literally part of* the delegate.

My goal was to find a theoretic foundation that is practically 
useful and doesn't defy expectations. For if a closure mutates a 
captured variable, one can't assign that closure to a `const` 
variable, notably, you cannot bind it to a functional's `const` 
parameter, I guess does defy expectations greatly.

Trying to draw a comparison with it, I found out today that 
slice's `capacity` is `pure` and also that it's a bug admitted in 
`object.d` ("This is a lie. [It] is neither `nothrow` nor `pure`, 
but this lie is necessary for now to prevent breaking code.")

> It's completely unsound, e.g., it allows creating race 
> conditions in `@safe` code.

Maybe I'm just too uncreative or too dumb to come up with one 
myself. I once ran into something like that trying out 
`std.parallelism.parallel` and how much it could gain me. It's 
years ago and I cannot remember a lot. I figured it wasn't 
applicable in my case. The

I'd really appreciate an example from your side.

> You can't say qualifiers are transitive except in this one 
> case, that translates to them not being transitive at all. A 
> lot of D's type system design is built on qualifiers being 
> transitive.
>
>> Since the language gives no non-UB way to assign the function 
>> pointer and the context pointer separately, it is not unsound.
>> [Here](https://forum.dlang.org/post/gauloixsonnnlswhbiqe@forum.dlang.org) is the space for discussing this issue. Unfortunately, it's mostly us two that care.
>
> The obfuscated DIP is unfortunately not helping with that.
>
> In any case, this is now the place to discuss this issue as you 
> have chosen to use this bug as a basis for evolving the 
> language.
>
>>> `toString` is `const`, `sink` is `const`, the only reference 
>>> to result accessible to `toString` is in the context of 
>>> `sink`, but somehow `result` is mutated anyway.
>>
>> See the paragraph above.
>>
>>> Unsoundness should be fixed, not embraced!
>>
>> Yes, I guess no one disagrees on that one, but on the question 
>> of it being an instance of it.
>>
>>> Finally, there's this concern: The DIP assumes that the only 
>>> reasonable way to manipulate delegates in higher-order 
>>> functions involves calling them, but this is not accurate.
>>
>> It assumes that the the most common use-case of non-mutable 
>> delegate parameters is only calling them. Returning them is 
>> another, but a rarer one. The DIP details that in this case, 
>> the author of the `compose` function needs to remember not to 
>> make the parameters mutable.
>
> I guess you mean the other way around.

Yes. I meant to say: "needs to remember to make the parameters 
mutable.".

>>> ```D
>>> auto compose(A,B,C)(C delegate(B) f, B delegate(A) g)pure{
>>>     return a=>f(g(a));
>>> }
>>> ```
>>>
>>> With the proposed changes, composing impure functions 
>>> suddenly becomes an impure operation as soon as you abstract 
>>> it into a higher-order function. This is pure nonsense. If 
>>> you have a `pure` expression and abstract it into a `pure` 
>>> function, it should not become less `pure` in the process!
>>
>> You did it correctly in the sense of the DIP. `compose` takes 
>> `f` and `g` as mutable. None of the proposed changes apply to 
>> mutable delegate parameters.
>
> Fair, but that's a technicality tangential to my point, and as 
> you may have been able to tell, I have certain reservations 
> about reusing `const` in this fashion.
>
>> By the changes proposed by this DIP, `compose` is `pure`. 
>> However, all delegates you pass to it lose information of 
>> attributes because you _could_ assign `f` or `g` in `compose`, 
>> no problem.
>
> But that's a terrible reason to not be able to annotate them 
> `const`. `const` means "this won't change", it does not mean 
> "if you compose this, it won't be recognized as `pure`" and 
> there is no clear way to get from one to the other. It's a 
> textbook example of a messy non-orthogonal design that does not 
> make any sense upon closer inspection.

Maybe use `in` (i.e. `const scope`) then? It clearly signifies: 
This is to read information from, not to assign to it, assign it 
to a global, not even to return it in any fashion.

>> But as you don't intend to mutate `f` or `g` in it, you could 
>> get the idea of making them `const` like this:
>
> Yes, let's assume that was my intention.
>
>> ```D
>> C delegate(A) compose(A, B, C)(const C delegate(B) f, const B 
>> delegate(A) g) pure
>> {
>>     return a => f(g(a));
>> }
>> ```
>> Then, by the proposed changes, only `pure` arguments lead to a 
>> `pure` call expression.
>
> Which was my point. This is indefensible.

It suffices to write this and one `@safe` unit test: The compile 
error will tell you there's a problem. I can add to the Error 
Messages section that in this case, the error message should hint 
that the `const` might be used improperly.

>> However, `compose` is a good example why this is not an issue: 
>> It is already a template. Why not go the full route and make 
>> the `delegate` part of the template type arguments like  this:
>> ```D
>> auto compose(F : C delegate(B), G : B delegate(A), A, B, C)(F 
>> f, G g) pure
>> {
>>     return delegate C(A arg) => f(g(arg));
>> }
>> ```
>
> The fact that there is some ugly workaround for my illustrative 
> example that also defeats the point of your DIP does not 
> eliminate the problem with the DIP.

This isn't an ugly workaround, but merely an attempt to stick to 
the example. Simply omitting the specialization syntax isn't 
possible. `return a => f(g(a));` doesn't compile, you need the 
`(A a)` part and for that, you need `A`. You can get it 
alternatively with `Parameters!f`; but `auto compose(F, G)(F f, G 
g)` with `return a => f(g(a));` doesn't work.

> Your reinterpretation of what delegate qualifiers mean would 
> need a DIP in its own right and it would hopefully be rejected.

I'm not sure it's a *re-*interpretation. As factually the 
compiler defines the language at places, you're probably right 
about the DIP part.