Compiler patch for runtime reflection

[Robert Jacques]

On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd at gmx.net> wrote:
> Am 25.10.11 16:41, schrieb Robert Jacques:
>> On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd at gmx.net> wrote:
>>> Hi,
>>>
>>> I would love to see runtime reflection be available in D. I am
>>> currently reading Walter Bright's "The D
>>> Programming Language" and with every page I read I became more and
>>> more fascinated and enthusiastic
>>> about D. But then I started to wonder why I haven't read about runtime
>>> reflection yet, so I did some Internet
>>> research and was a bit disappointed this feature isn't available. D is
>>> really cool, it's not only C done right,
>>> but also Java and C#. Actually, to me, it's one of the best
>>> programming language available yet. It's really a
>>> pitty RTTI is not available. So I hope your approach will somehow
>>> influence D's future :)
>>>
>>> Keep going,
>>> Jonny
>>
>> Hi Jonny,
>> Well, some RTTI is available. How much RTTI is needed, really depends on
>> what you're trying to do. Therefore whether D has or hasn't got RTTI
>> depends heavily on what any particular person means by RTTI. Do you have
>> some use cases in mind? Maybe prototype objects, duck-typing/casting or
>> serialization? Or perhaps something you wrote?
>>
>> Sorry for all the questions, but I feel as if we don't have a resident
>> reflection expert in the dialog, or even the opinions of
>> basic/experienced users. We all know that 'RTTI' is good, but we're a
>> little vague on the how and why. (Well, beyond the use cases above)
>
> Hi Robert,
>
> Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
> "
> [...]
> - Discover and modify source code constructions (such as code blocks,
> classes, methods, protocols, etc.) as a first-class object at runtime.
> - Convert a string matching the symbolic name of a class or function
> into a reference to or invocation of that class or function.
> [...]
> "
>
> Here is what I would dream of for arbitrary objects/classes (not
> necessarily known at compile-time):
> - Query an object for its list of methods together with their
> signatures. Select a method, bind some values to its arguments, call it,
> and retrieve the return type (if any).
> - Query an object for its public fields (at least), and provide a way to
> get/set their values.
> - Query an object's class for all implemented interfaces and its base class.
> - Query a module for all type definitions and provide a way to
> introspect these types in more detail. For instance, it would be really
> cool if I could find a class with name "Car" in module "cars", get a
> list of all defined constructors, select one, bind values to the
> constructor's parameters, and create a corresponding object.
>
> Now you might ask where is the use case. I think, a very important one,
> besides creating GUI designers, is to be able to implement a Dependency
> Injection container [2,3,4] like, for example, the one provided by the
> Spring Framework for Java or .NET. In short, they allow you define how
> object trees should be build up solely by specifying this tree in an XML
> file. Within the XML file you can create new instances of classes of
> arbitrary types. You can specify which constructors to use for
> instantiation, and what values the constructor's arguments (if any)
> should have. You can even bind such an argument to an object previously
> created by the XML definition. You can call methods on existing objects,
> e.g. in order to call setter to further initialize an object with values
> you also define in that XML file. That XML file can be loaded when your
> program starts an make the DI container component instantiate all needed
> objects for you. There is no need for recompilation if the XML file is
> changed. Just restart your application.
>
> Implementing such a DI container heavily depends on reflection, because
> the DI container component doesn't know anything about the objects to be
> created during runtime.
>
> Qt also extends C++ with a reflection mechanism through the help of its
> meta object compiler (moc). It analyses the C++ source code, generates
> meta class definitions [6,7] and weaves them into your Qt class. Hence,
> in Qt, you can query an object for fields, methods, interfaces, etc. and
> you can call methods with arbitrary parameters, or you can instantiate a
> class using an arbitrary constructor. Consequently, somone implemented a
> DI container for C++ which is based on Qt and works more or less the
> same way the Spring DI container does. You can build up object trees
> simply by specifying such trees in an XML file.
>
> I don't go into why dependency injection is a very powerful feature.
> This is Martin Fowler's [3] job ;) But when I program with C++ I miss
> such a flexible dependency injection mechanism a lot. And I hope this
> will eventually be available for D.
>
> Cheers,
> Jonny
>
> [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
> [2] http://en.wikipedia.org/wiki/Dependency_injection
> [3] http://martinfowler.com/articles/injection.html
> [4]
> http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
> [5] http://qtioccontainer.sourceforge.net/
> [6] http://doc.qt.nokia.com/stable/qmetaobject.html
> [7]
> http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx

Hi Jonny,
Thank you for your informative (and well cited) post. It has provided me with a new take on an old design pattern and some enjoyable reading. In return, let me outline my opinion of reflection in D today, and tomorrow, as it pertains to your wish list.

Reflection in D today is very different from the host of VM languages that have popularized the concept. Being a compiled systems language, actual runtime self-modification is too virus like to become at a language level feature. However, given the compilation speed of D, people have made proof of concept libraries that essentially wrapped the compiler and dynamically loaded the result. As LDC uses LLVM, which has a jit backend, I'd expect to see something get into and D 'eval' library into etc eventually. (phobos uses the BOOST license, which isn't compatible with LLVM).

Compile-time reflection and generation of code, on the other hand, is something D does in spades. It fulfills your dream list, although I think module level reflection might only be available in the github version. The API design is still in flux and we are actively iterating / improving it as find new uses cases and bugs. The current plan is to migrate all the traits functions over to a special 'meta' namespace (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid libraries for each of the concepts I listed, (prototype objects, duck-typing/casting or serialization), have been written using the compile-time meta-programming features in D. So that's the good.

On the other hand, D's runtime capabilities are limited to object.factory, the under implemented RTTI and library solutions which manually expose information gathered by D's compile-time mechanisms. And so far, these tools have been more than enough, from a functionality point of view. Most of our desire for better runtime reflection stems from a desire for efficiency, composition, cleanliness of user syntax and simplification of library code. These are all important issues for the widespread use of reflection based libraries, but they're not 'I can't implement X' issues.

As for the future, I believe that the division in D between compile-time and run-time reflection warrants a serious look at the design of the run-time half of the system. To that end, I believe that implementing reflection in a library would be the best way to experiment and iterate an API. To that end, I have a proposal in the review queue to improve std.variant which contains dynamic dispatch (i.e. the ability to get/set public fields and call methods), duck-typeing/casting and prototype-style objects. Notably, it's missing introspection capabilities as thats what I'm most unsure about API wise, and simplest to add. Designing reflection inside a library keeps Walter & Co's bandwidth free for other bugs/features and provides a very good stress test of D's meta-programming capabilities. (Both of which I think are good things)
(I am soliciting feedback, if you care to take a look: https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)

And there are many reasons we might want to experiment with D's runtime reflection API instead of just copying someone. For example, take Dependency Injection. Using Java-style reflection, DI suffers from the Reflection injection and Unsafe Reflection security vulnerabilities. To do DI safely, you have to validate your inputs and D's compile-time reflection provides a perfect way to implement validated DI. Every time I hear about some form of injection, be it SQL or JSON or Reflection, hit the news, makes me think that 5-10 years from now well look back on the use of unvalidated dynamic code constructs the same way we do about null terminated arrays today.