First-class polymorphic lamdas?

[Vladimir Panteleev via Digitalmars-d]

On Friday, 5 February 2016 at 20:36:20 UTC, ZombineDev wrote:
> In D, on the other hand, the compiler stores a reference to the 
> lambda function as a function-pointer (or a delagate-pointer, 
> if any local variables are captured), however this precludes 
> support for polymorhpic lambdas. AFAIU, delegates are 
> implemented the way they are because this enables much easier 
> ABI interoperability.

AIUI, what's happening underneath, is that any time something 
needs a context pointer, it is "nested" under it, and its "this" 
becomes the context. I.e.:

void main()
{
     int n;
     alias dg = x => n+=x;
}

This is obviously equivalent to:

void dg(T)(T x) { n += x; }

i.e. dg is a template of a local function. Obvious so far. 
However, if next you do:

void callIt(alias fun)()
{
     fun(5);
}

callIt!dg();

callIt becomes nested under main(), i.e. its context pointer will 
be main's stack frame. This precludes callIt already having a 
context pointer, which is why you can't use methods together with 
alias parameters to symbols with context, or multiple alias 
parameters to things with different context. AIUI SDC doesn't 
have this limitation. Anyway, you probably knew all this already.

> Kenji worked on supporting polymorphic lambdas for DMD 2.070 
> [4][5] and his design
> (if I understand it correctly) works like so:
> ```
> alias add = (a,b) => a + b;
> // is syntactically lowered to
>
> alias add = add__compile_generated;
> template add__compile_generated(T, U)
> {
>     auto add__compile_generated(T a, U b) { return a + b; }
> }
> ```

I believe the 2.070 changes are entirely confined to syntax.

Previously, the syntax:

alias add = (a,b) => a + b;

would not be valid. I mentioned this (and the workaround) in a 
blog post a while ago:

https://blog.thecybershadow.net/2015/04/28/the-amazing-template-that-does-nothing/

and briefly at DConf IIRC. TL;DR: The solution was to wrap it in 
a template which aliases to itself:

alias I(alias X) = X;
alias add = I!((a,b) => a + b);

> auto map(alias fn, Rf)(Rf step)
> {
>     struct MapResult
>     {
>         Rf step;
>
>         auto opCall(S, Input...)
>             (auto ref S state, auto ref Input inputs)
>         {
>             return step(state, fn(inputs));
>         }
>     }
>     MapResult result = { step : step };
>     return result;
> }

For some reason I didn't realize you can also create a value type 
templated functor by using explicitly nested structs. I had 
stumbled upon the new feature from 
https://github.com/dlang/dmd/pull/5518 by accident while working 
on something similar (heterogenous iteration and range-like 
composition, e.g. over struct fields). Will post that here later 
today.

> To sum up, I want the compiler to do all those things for me, 
> like the C++14 compilers do. Presently it feels like I'm doing 
> the compiler's job.

I think that alias template parameters and context binding are a 
very rich and unexplored area of D, which will allow writing some 
really efficient and composable code. A while ago I created a few 
DMD pull requests with related improvements, and based on those 
DMD pull requests I wrote an allocator and serialization library. 
The entire code was messy but extremely efficient (e.g. the 
"this" pointer was shared between all composed layers of the 
entire allocator instance).

DMD pull requests:

https://github.com/dlang/dmd/pull/3329
https://github.com/dlang/dmd/pull/3345
https://github.com/dlang/dmd/pull/3361

Compiler capability tests:

https://github.com/CyberShadow/ae/blob/master/utils/meta/caps.d

Allocators (note that this predates std.experimental.allocator):

https://github.com/CyberShadow/ae/blob/master/utils/alloc.d

Serialization:

https://github.com/CyberShadow/ae/tree/master/utils/serialization

For personal use, I still use a hacked compiler with 
https://github.com/dlang/dmd/pull/3884 reverted, even though at 
the time I argued in favor of that change (since the introduction 
of field context binding broke code). It is just too useful.

A while ago Kenji proposed "static alias", to explicitly control 
whether an alias argument passes context or not (sometimes not 
passing context is required, e.g. if the parameter will be used 
only for introspection or with my __traits(child)):

https://wiki.dlang.org/Brush_Up_Language_Features#Nested_Symbols

I think by now we would need "this alias" since fields and 
methods no longer bind with context to alias parameters.

I would like to some day wrap all of the above work together into 
a coherent proposal to improve D aliases, but haven't had the 
time/energy so far (few people seem to share my enthusiasm for 
the subject :)). Perhaps for DConf 2017...