DeRailed DSL (was Re: compile-time regex redux)

[Andrei Alexandrescu (See Website For Email)]

Kirk McDonald wrote:
> Bill Baxter wrote:
>> Speaking of which I'm surprised Kirk hasn't piped in here more about 
>> how this could make life easier for PyD (or not if that's the case).  
>> Any thoughts, Kirk?  You're in one of the best positions to say what's 
>> a bottleneck with the current state of compile-time reflection.
>>
>> --bb
> 
> One area of Pyd which I am unhappy with is its support for inheritance 
> and polymorphic behavior.
> 
> http://pyd.dsource.org/inherit.html

Great lib, and a good place to figure how introspection can help.

> Getting the most proper behavior requires a bit of a workaround. For 
> every class that a user wishes to expose to Python, they must write a 
> "wrapper" class, and then expose both the wrapper and the original class 
> to Python. The basic idea is so that you can subclass D classes with 
> Python classes and then get D code to polymorphically call the methods 
> of the Python class:
> 
> // D class
> class Foo {
>     void bar() { writefln("Foo.bar"); }
> }
> 
> // D function calling method
> void polymorphic_call(Foo f) {
>     f.bar();
> }
> 
> # Python subclass
> class PyFoo(Foo):
>     def bar(self):
>         print "PyFoo.bar"
> 
> # Calling D function with instance of Python class
>  >>> o = PyFoo()
>  >>> polymorphic_call(o)
> PyFoo.bar
> 
> Read that a few times until you get it. To see how Pyd handles this, 
> read the above link. It's quite ugly.

If I understand things correctly, in the ideal setup you'd need a means
to expose an entire, or parts of, a class to Python. That is, for the class:

class Base {
     void foo() { writefln("Base.foo"); }
     void bar() { writefln("Base.bar"); }
}

instead of (or in addition to) the current state of affairs:

wrapped_class!(Base) b;
b.def!(Base.foo);
b.def!(Base.bar);
finalize_class(b);

it would be probably desirable to simply write:

defclass!(Base);

which automa(t|g)ically takes care of all of the above.

To do so properly, and to also solve the polymorphic problem that you
mention, defclass must define the following class:

class BaseWrap : Base {
     mixin OverloadShim;
     void foo() {
         get_overload(&super.foo, "foo");
     }
     void bar() {
         get_overload(&super.bar, "bar");
     }
}

Then the BaseWrap (and not Base) class would be exposed to Python, along
with each of its methods.

If I misunderstood something in the above, please point out the error
and don't read the rest of this post. :o)

This kind of task should be easily doable with compile-time reflection,
possibly something along the following lines (for the wrapping part):

class PyDWrap(class T) : T
{
   mixin OverloadShim;
   // Escape into the compile-time realm
   mixin
   {
     foreach (m ; methods!(T))
     {
       char[] args = formals!(m).length
         ? ", " ~ actuals!(m) : "";
       writefln("%s %s(%s)
         { return get_overload(super.%s, `%s`%s); }",
         ret_type!(m), name!(m), formals!(m),
         name!(m), name!(m), args);
     }
   }
}

So instantiating, say, PyDWrap with Base will tantamount to this:

class PyDWrap!(Base) : Base {
     mixin OverloadShim;
     void foo() {
         return get_overload(&super.foo, `foo`);
     }
     void bar() {
         get_overload(&super.bar, `bar`);
     }
}

Instantiating PyDWrap with a more sophisticated class (one that defines
methods with parameters) will work properly as the conditional
initialization of args suggests.

> The D wrapper class for Foo would look something like this:
> 
> class FooWrapper : Foo {
>     mixin OverloadShim;
>     void bar() {
>         get_overload(&super.bar, "bar");
>     }
> }
> 
> Never mind what this actually does. The problem at hand is somehow 
> generating a class like this at compile-time, possibly given only the 
> class Foo. While these new mixins now give me a mechanism for generating 
> this class, I don't believe I can get all of the information about the 
> class that I need at compile-time, at least not automatically. I might 
> be able to rig something creative up with tuples, now that I think about 
> it...

At the end of the day, without compile-time introspection, code will end
up repeating itself somewhere. For example, you nicely conserve the
inheritance relationship among D classes in their Python incarnations.
Why is that possible? Because D offers you the appropriate introspection
primitive. If you didn't have that, or at least C++'s SUPERSUBCLASS
trick (which works almost by sheer luck), you would have required the
user to wire the inheritance graph explicitly.

> However, I have some more pressing issues with Pyd at the moment
> (strings, embedding, and building, for three examples), which have
> nothing to do with these new features.

Update early, update often. :o) Please write out any ideas or issues you
are confronting. Looks like PyD is a good case study for D's nascent
introspection abilities.


Andrei