Can D do some sort of automatic template to "dynamic template" wrapping

[Adam via Digitalmars-d]

On Saturday, 19 September 2015 at 15:56:23 UTC, Laeeth Isharc 
wrote:
> On Friday, 18 September 2015 at 20:55:06 UTC, Adam wrote:
>> Fact: Templates are static!
>>
>> But templates are still normal functioning code and can 
>> actually work dynamically. We can see this by including a D 
>> compiler inside an app and have the app re-compile the 
>> templates for specific dynamic types. It doesn't need 
>> templates to do of course, but it shows that "Templates are 
>> static" is not quite as simple as it sounds.
>>
>> But could D actually create some sort of internal expression 
>> of D templates as dynamic objects that can be used in run-time?
>>
>> If have a template T that only uses 3 types, then one can 
>> simple do something like(pseudo):
>>
>> t(object o)
>> {
>> // use dynamic reflection or whatever to get the type of the 
>> object o
>> if (o.type == 'int')
>>     return T!(int)(o.value, o.unit);
>> if (o.type == 'float')
>>     return T!(float)(o.value, o.unit);
>> return T!(unknown)(o.value, o.unit);
>> }
>>
>> etc...
>>
>> t then is a "dynamic representation" of the static 
>> functionality of T.
>>
>> But my guess is that the compiler could essentially do this 
>> for us and allow one to use templates in a dynamic way. Most 
>> of the time it may not be of much use but, say, if one is 
>> interacting with console and T converts a string to a number. 
>> T could convert a number with unit such as "3mm" in to some 
>> base representation such as 0.003(m). float's are handled 
>> differently than ints(rounding depending on some criteria).
>>
>> Maybe we use the T internally and its lightning fast. But for 
>> the console case, we  have write a dynamic version of the 
>> template or create something similar to the code above.
>>
>> But the compiler already knows, at least, some of the times 
>> that will be used and it should be able to instantiate the 
>> template for each of those types.
>>
>> e.g., if you use T!(float) in the code, it will know to create 
>> the "dynamic wrapper" to include that type in the list. If you 
>> don't every use T!(float) explicitly then you can simply 
>> create a useless statement like {auto x = 
>> T!(float)("3.04mm");} which will be optimized but the compiler 
>> will then include float as a type used in the dynamic version 
>> of T. I may not be making much sense here, but it's the best 
>> I'll do.
>>
>>
>>
>> The compiler then can do something like assemble the dynamic 
>> version of the template into a new function which can be used 
>> at compile time on any object.
>> Using a simple special character could inform the compiler to 
>> use the dynamic template version.
>>
>> object u = userInput();
>> &T(u); // (instead of using t(u)).
>>
>>
>> Here the compiler does all the work for us. If u has type 
>> float then this gets effectively dispatched to 
>> T!(float)(cast(float)u). If it's an int, then 
>> T!(int)(cast(int)u).
>>
>> What it mainly does for us is simplify having to do the cast 
>> our self along with the string of if checking for all the 
>> types or having to code that works dynamically
>>
>> Or is it impossible for the D compiler to accomplish such a 
>> task automatically for us?
>
> I think you could also do this using structs with a type enum 
> inside and a union for the data contents.  Then you can use 
> string mixins using traits to create generic runtime functions 
> that dispatch to the appropriate method/free standing function 
> on the underlying type.  Take a look at Algebraic and Adam 
> Ruppe's jsvar.
>
> Some of the genericity can be handled at compile time.  For 
> example, I have a templated time series entry struct.  A time 
> series is a slice of such structs.  An entry definitively has a 
> date, and it may have either a bid and ask or (a close and 
> optionally open, high, low, adjusted close).  It has a volume, 
> and may have an open interest.  The date may be a 
> year/month/day or it might have a time too.  I want to store 
> things efficiently in memory and on disk.  So I use 
> hasMember!(typeof(pricebar.date),"hour") to find out if it's an 
> intraday date or daily for example.  See Andrei's allocator 
> talk, his code, and maybe that for cerealed.
>
> I have a bunch of templated functions that can do useful things 
> with such a time series.
>
> But then I want to pass them around wrapped in an a tagged 
> union and generate versions of these functions that can operate 
> on this untemplated struct.  (I will wrap the slices, not the 
> entries).  That will make it easier to access my work from a 
> dynamic language like Lua.  I haven't written this bit yet, but 
> I have done something similar in another context.  A degree of 
> misery in trying to figure out std.traits, but that's a one off 
> cost and it goes quicker if you read others code for 
> inspiration.  UDAs help in specifying which functions should 
> have wrapped versions generated etc.

I'm thinking I've probably been miunderstood here. Let me clarify 
what I'm talking about and see if it still more sense.


Templated functions's types are resolved at compile time, so

void foo(T)(T x){}

is a templated function that operates on a type T and is a sort 
of compile time construct(since it depends on the unknown T). I 
do not mean it only can be used at compile time, but T must be 
known then.

So, foo!T only exists at compile time, right? Yet, for any actual 
type such as an int, foo!int is a runtime function(it exists in 
the binary).

The compiler figures out all the types used at compile time and 
creates all the various foo!N's. If K is the set of all the types 
used at compile type with foo, we could effectively write foo!K 
as a sort of collection of all the different foo's(but it's not 
really any different than foo!T except T is any type while K is 
the actual types used by the programmer).

Ok so far?

So, if the programmer called foo!int, foo!float in the code, the 
compiler then constructs

foo!K = {foo!int, foo!float} and puts them in the binary.

Then these are actual run-time functions and foo!K represents 
them. foo!K is as real as foo!T is, but restricted to only 
contains the actual used types and put in the binary. If the 
compiler could hypothetically put all of foo!T then the binary 
would be infinite, or, at least very large. So, instead it just 
keeps foo!K. If you want to expand K at run-time, you are out of 
luck.


Ok so far?

we can actually create foo!K easily(assuming foo!T(int))

fooK(string K, object i)
{
     if (K == "int")
          foo!int(i);
     else if (K == "float")
          foo!float(i);
     else
          foo!unknown(i);    // unknown is special, not needed but 
it would only add one element to foo!K, so the binary would not 
be much bigger and we could specialize foo for "unknown" to 
handle those cases such as throwing an exception.
}

Ok?

now fooK is a function that exists that works at run-time on all 
types. It is an approximation to foo!T(int).

If, say, in your code, you now use foo!string, then on recompile 
we would have

fooK(string K, object i)
{
     if (K == "int")
          foo!int(i);
     else if (K == "float")
          foo!float(i);
     else if (K == "string")
          foo!string(i);
     else
          foo!unknown(i);
}

Again, we can do this all by hand. And we can call fooK on 
objects at run-time rather than known types at compile time.


The only reason we can't use foo!T is because the compiler 
doesn't know all the types we will be using(at run-time), if it 
did, we could use foo!T at run-time. It simply does the best job 
it can and gets the compile time uses. It can't do anything about 
run-time, I guess, unless it dynamically built the functions at 
run-time, Which is a possibility I suppose but it would require a 
D compiler inside the code to create the new instance of the 
template for the new time.


fooK though works like any other non-templated function, so it 
can be used at run time for us.

fooK can be generated for us by the compiler if it was "smart 
enough". It could add type checking and such to remove the string 
K parameter if it can find out what the type of i is. e.g., use 
reflection for something like

fooK(object i)
{
     string K = typeof(i).stringof;
     if (K == "int")
          foo!int(i);
     else if (K == "float")
          foo!float(i);
     else if (K == "string")
          foo!string(i);
     else
          foo!unknown(i);
}


and, in fact, one could get fancy and use a hash table instead of 
the if's. Then this would allow one to extend fooK at run-time 
sort of like oop. (just add your new foo!X to the table and it 
will be called)

The problem for the programmer is that when the number of 
parameters, the number of if statements grow by #K^n. For 1 
parameter it's #K and for 2 it's #K^2 since you'll have to check 
for all pairs of types. It also makes it hard to maintain the 
above for new added types. You'll have to do it for every 
templated function and update all of them every time a new 
specialization is added.

Ok so far?


What's cool, if all this makes sense. If fooK is used on "compile 
time types", it can be optimized out to the static call. So fooK 
could always be used(we would want to use foo!K for notation, of 
course) and would function as both a compile time type of 
templated function and also a run-time function.

If foo was implemented with a look up table, then templated 
functions could be added at run time to extend the behavior. 
E.g., a plugin could could add a "type handler" to foo!K to 
handle some new type it's using.

Ultimately, what would happen is that templated functions would 
be "first class citizens" and work not only in compile type but 
at run-time too. It sort of combines both compile-time 
reflection(what templates do now) and run-time reflection(what 
that don't) and the compiler figures it out and keeps track of it 
all. For the programmer, there would be no difference except for 
the unknown case. Performance wise, there should be no 
difference. At compile time, the compiler would do pretty much 
exactly what it does now, just use the explicitly known type. It 
would have the same performance as before. But when it can't do 
this(the types are unknown), it falls back onto the runtime 
lookup version.

Does any of that make sense? ;)

The reason one can't implement this as a library function, as far 
as I know, is because one would have to know all the types the 
programmer calls foo!T with at "compile-time"(e.g., we would have 
to have a K, the set of those types). If one could get that 
information then fooK could be automatically constructed using 
string mixins. Not sure if optimization would take place, I 
suppose some static checking could be used to get the 
compile-time types then a normal lookup table for the others.

If such a fooK was known, then programmer would always use that 
instead and be confident that the compiler would always choose 
the best situation(if it can use a direct call to the foo!N for 
some N, then it uses that, if not, it goes through the lookup 
process).

I am making one big assumption here, and that is one can get the 
type of an object at run-time. I'm not sure if this is possible 
in D and there maybe other issues with inheritance and such which 
might cause problems with more complicated cases than I've 
assumed here.

What's more, is with fooK, templated functions can take place as 
virtual functions! Since fooK encomposes both compile time and 
dynamic behavior in one) Of course, when used as virtual 
functions, unless in certain cases when optimization can take 
place, the run-time behavior will always be used.

I doubt anyone would read all so I'll stop now ;)