DeRailed DSL (was Re: compile-time regex redux)

[Kirk McDonald]

Andrei Alexandrescu (See Website For Email) wrote:
> Kirk McDonald wrote:
>> Bill Baxter wrote:
>>> Speaking of which I'm surprised Kirk hasn't piped in here more about 
>>> how this could make life easier for PyD (or not if that's the case).  
>>> Any thoughts, Kirk?  You're in one of the best positions to say 
>>> what's a bottleneck with the current state of compile-time reflection.
>>>
>>> --bb
>>
>> One area of Pyd which I am unhappy with is its support for inheritance 
>> and polymorphic behavior.
>>
>> http://pyd.dsource.org/inherit.html
> 
> Great lib, and a good place to figure how introspection can help.

Thanks!

> 
>> Getting the most proper behavior requires a bit of a workaround. For 
>> every class that a user wishes to expose to Python, they must write a 
>> "wrapper" class, and then expose both the wrapper and the original 
>> class to Python. The basic idea is so that you can subclass D classes 
>> with Python classes and then get D code to polymorphically call the 
>> methods of the Python class:
>>
>> // D class
>> class Foo {
>>     void bar() { writefln("Foo.bar"); }
>> }
>>
>> // D function calling method
>> void polymorphic_call(Foo f) {
>>     f.bar();
>> }
>>
>> # Python subclass
>> class PyFoo(Foo):
>>     def bar(self):
>>         print "PyFoo.bar"
>>
>> # Calling D function with instance of Python class
>>  >>> o = PyFoo()
>>  >>> polymorphic_call(o)
>> PyFoo.bar
>>
>> Read that a few times until you get it. To see how Pyd handles this, 
>> read the above link. It's quite ugly.
> 
> If I understand things correctly, in the ideal setup you'd need a means
> to expose an entire, or parts of, a class to Python. That is, for the 
> class:
> 
> class Base {
>     void foo() { writefln("Base.foo"); }
>     void bar() { writefln("Base.bar"); }
> }
> 
> instead of (or in addition to) the current state of affairs:
> 
> wrapped_class!(Base) b;
> b.def!(Base.foo);
> b.def!(Base.bar);
> finalize_class(b);
> 
> it would be probably desirable to simply write:
> 
> defclass!(Base);
> 
> which automa(t|g)ically takes care of all of the above.
> 

That would be nice. However, my gut (and previous experience) tells me 
that it is simpler, or at least more reliable, if the user explicitly 
lists the things to wrap. There remain bits and pieces of D that I can't 
expose to Python, and it's easier to allow the user to simply not 
specify those things, than to detect and not wrap them.

This is moot if D's reflection becomes perfect. We are not there, yet, 
however, so I must play with what we have.

> To do so properly, and to also solve the polymorphic problem that you
> mention, defclass must define the following class:
> 
> class BaseWrap : Base {
>     mixin OverloadShim;
>     void foo() {
>         get_overload(&super.foo, "foo");
>     }
>     void bar() {
>         get_overload(&super.bar, "bar");
>     }
> }
> 
> Then the BaseWrap (and not Base) class would be exposed to Python, along
> with each of its methods.
> 
> If I misunderstood something in the above, please point out the error
> and don't read the rest of this post. :o)

One minor nit: Both BaseWrap and Base must be wrapped by Pyd, although 
only BaseWrap will actually be exposed to Python. (Meaning Python code 
can subclass and create instances of BaseWrap, but not Base.) This is so 
that D functions can return instances of Base to Python.

> 
> This kind of task should be easily doable with compile-time reflection,
> possibly something along the following lines (for the wrapping part):
> 
> class PyDWrap(class T) : T
> {
>   mixin OverloadShim;
>   // Escape into the compile-time realm
>   mixin
>   {
>     foreach (m ; methods!(T))
>     {
>       char[] args = formals!(m).length
>         ? ", " ~ actuals!(m) : "";
>       writefln("%s %s(%s)
>         { return get_overload(super.%s, `%s`%s); }",
>         ret_type!(m), name!(m), formals!(m),
>         name!(m), name!(m), args);
>     }
>   }
> }
> 
> So instantiating, say, PyDWrap with Base will tantamount to this:
> 
> class PyDWrap!(Base) : Base {
>     mixin OverloadShim;
>     void foo() {
>         return get_overload(&super.foo, `foo`);
>     }
>     void bar() {
>         get_overload(&super.bar, `bar`);
>     }
> }
> 
> Instantiating PyDWrap with a more sophisticated class (one that defines
> methods with parameters) will work properly as the conditional
> initialization of args suggests.
> 

My current idea, which can be done right now, involves some simple 
refactoring of the class-wrapping API, which was designed before we had 
proper tuples. It would look something like this:

wrap_class!(
     Base,
     Def!(Base.foo)
     Def!(Base.bar),
);

'Def' would become a struct or class template. (The capital 'D' 
distinguishes it from the 'def' function template used to wrap regular 
functions.) Since all of the methods are now specified at compile-time, 
I can generate the wrapper class at compile time with little difficulty.

>> The D wrapper class for Foo would look something like this:
>>
>> class FooWrapper : Foo {
>>     mixin OverloadShim;
>>     void bar() {
>>         get_overload(&super.bar, "bar");
>>     }
>> }
>>
>> Never mind what this actually does. The problem at hand is somehow 
>> generating a class like this at compile-time, possibly given only the 
>> class Foo. While these new mixins now give me a mechanism for 
>> generating this class, I don't believe I can get all of the 
>> information about the class that I need at compile-time, at least not 
>> automatically. I might be able to rig something creative up with 
>> tuples, now that I think about it...
> 
> At the end of the day, without compile-time introspection, code will end
> up repeating itself somewhere. For example, you nicely conserve the
> inheritance relationship among D classes in their Python incarnations.
> Why is that possible? Because D offers you the appropriate introspection
> primitive. If you didn't have that, or at least C++'s SUPERSUBCLASS
> trick (which works almost by sheer luck), you would have required the
> user to wire the inheritance graph explicitly.
> 
>> However, I have some more pressing issues with Pyd at the moment
>> (strings, embedding, and building, for three examples), which have
>> nothing to do with these new features.
> 
> Update early, update often. :o) Please write out any ideas or issues you
> are confronting. Looks like PyD is a good case study for D's nascent
> introspection abilities.
> 

It has always been one. Pyd was probably the first D library to have a 
serious need for tuples, going so far as to fake them before they were 
part of the language proper. It's probably the largest concrete 
application of meta-programming written in D.

-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org