Compiler patch for runtime reflection

[Jonny Dee]

Hello Robert,

Am 26.10.11 07:16, schrieb Robert Jacques:
> On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd at gmx.net> wrote:
>> Am 25.10.11 16:41, schrieb Robert Jacques:
>>> On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd at gmx.net> wrote:
>>>> [...]
>> Hi Robert,
>>
>> Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
>> "
>> [...]
>> - Discover and modify source code constructions (such as code blocks,
>> classes, methods, protocols, etc.) as a first-class object at runtime.
>> - Convert a string matching the symbolic name of a class or function
>> into a reference to or invocation of that class or function.
>> [...]
>> "
>>
>> Here is what I would dream of for arbitrary objects/classes (not
>> necessarily known at compile-time):
>> - Query an object for its list of methods together with their
>> signatures. Select a method, bind some values to its arguments, call it,
>> and retrieve the return type (if any).
>> - Query an object for its public fields (at least), and provide a way to
>> get/set their values.
>> - Query an object's class for all implemented interfaces and its base
>> class.
>> - Query a module for all type definitions and provide a way to
>> introspect these types in more detail. For instance, it would be really
>> cool if I could find a class with name "Car" in module "cars", get a
>> list of all defined constructors, select one, bind values to the
>> constructor's parameters, and create a corresponding object.
>>
>> [...]
>>
>> Implementing such a DI container heavily depends on reflection, because
>> the DI container component doesn't know anything about the objects to be
>> created during runtime.
>>
>> Qt also extends C++ with a reflection mechanism through the help of its
>> meta object compiler (moc). It analyses the C++ source code, generates
>> meta class definitions [6,7] and weaves them into your Qt class. Hence,
>> in Qt, you can query an object for fields, methods, interfaces, etc. and
>> you can call methods with arbitrary parameters, or you can instantiate a
>> class using an arbitrary constructor. Consequently, somone implemented a
>> DI container for C++ which is based on Qt and works more or less the
>> same way the Spring DI container does. You can build up object trees
>> simply by specifying such trees in an XML file.
>>
>> I don't go into why dependency injection is a very powerful feature.
>> This is Martin Fowler's [3] job ;) But when I program with C++ I miss
>> such a flexible dependency injection mechanism a lot. And I hope this
>> will eventually be available for D.
>>
>> Cheers,
>> Jonny
>>
>> [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
>> [2] http://en.wikipedia.org/wiki/Dependency_injection
>> [3] http://martinfowler.com/articles/injection.html
>> [4]
>> http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
>>
>> [5] http://qtioccontainer.sourceforge.net/
>> [6] http://doc.qt.nokia.com/stable/qmetaobject.html
>> [7]
>> http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx
>>
>
> Hi Jonny,
> Thank you for your informative (and well cited) post. It has provided me
> with a new take on an old design pattern and some enjoyable reading. In
> return, let me outline my opinion of reflection in D today, and
> tomorrow, as it pertains to your wish list.

Many thanks to you, too, for your very elaborate answer :)

> Reflection in D today is very different from the host of VM languages
> that have popularized the concept. Being a compiled systems language,
> actual runtime self-modification is too virus like to become at a
> language level feature. However, given the compilation speed of D,
> people have made proof of concept libraries that essentially wrapped the
> compiler and dynamically loaded the result. As LDC uses LLVM, which has
> a jit backend, I'd expect to see something get into and D 'eval' library
> into etc eventually. (phobos uses the BOOST license, which isn't
> compatible with LLVM).

I know, that "runtime self-modification" and runtime code generation is 
a "dangerous" feature. And there really are rare cases where using such 
an approach might justify the risc in using it. Although this feature is 
not on my wish list, it might be good for generating dynamic proxies to 
arbitrary object instances like they are used by some ORMs. See 
Hibernate/NHibernate, for example [1,2]. Another example is 
aspect-oriented programming. But while I can't see the exacty reason for 
it, such a feature might indeed be a feature which is more appropriate 
for VM languages.

> Compile-time reflection and generation of code, on the other hand, is
> something D does in spades. It fulfills your dream list, although I
> think module level reflection might only be available in the github
> version. The API design is still in flux and we are actively iterating /
> improving it as find new uses cases and bugs. The current plan is to
> migrate all the traits functions over to a special 'meta' namespace
> (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
> libraries for each of the concepts I listed, (prototype objects,
> duck-typing/casting or serialization), have been written using the
> compile-time meta-programming features in D. So that's the good.

A absolutely agree! D's compile-time reflection is very good. And it's 
hard to top this.

> On the other hand, D's runtime capabilities are limited to
> object.factory, the under implemented RTTI and library solutions which
> manually expose information gathered by D's compile-time mechanisms. And
> so far, these tools have been more than enough, from a functionality
> point of view. Most of our desire for better runtime reflection stems
> from a desire for efficiency, composition, cleanliness of user syntax
> and simplification of library code. These are all important issues for
> the widespread use of reflection based libraries, but they're not 'I
> can't implement X' issues.

You are right, there IS certainly always some way to 'implement X'. But, 
as you know, there is always a consideration of the effort you need to 
implement X.

> As for the future, I believe that the division in D between compile-time
> and run-time reflection warrants a serious look at the design of the
> run-time half of the system. To that end, I believe that implementing
> reflection in a library would be the best way to experiment and iterate
> an API. To that end, I have a proposal in the review queue to improve
> std.variant which contains dynamic dispatch (i.e. the ability to get/set
> public fields and call methods), duck-typeing/casting and
> prototype-style objects. Notably, it's missing introspection
> capabilities as thats what I'm most unsure about API wise, and simplest
> to add. Designing reflection inside a library keeps Walter & Co's
> bandwidth free for other bugs/features and provides a very good stress
> test of D's meta-programming capabilities. (Both of which I think are
> good things)

I've got no problem with an approach which puts runtime reflection 
capabiities into a separate library. No matter were you look at, Java, 
C#, or Qt, all have a library for reflection purpose. I do not see, 
however, how this might be done without compiler support. As already 
mentioned, Qt has its own moc compiler, which parses your C++ source 
code and generates the necessary infra structure. I'm still a beginner 
with respect to D, so I don't know what is really already possible, and 
what not. But my current feeling is that a similar approach would also 
be needed for D. If the D compiler itself, or another post-compiler 
component should generate the meta information for runtime reflection is 
another question. I could live with both, although I'd prefer the former 
built-in one.

> (I am soliciting feedback, if you care to take a look:
> https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)

Thanks for this link, I'll certainly have a look at it.

> And there are many reasons we might want to experiment with D's runtime
> reflection API instead of just copying someone. For example, take
> Dependency Injection. Using Java-style reflection, DI suffers from the
> Reflection injection and Unsafe Reflection security vulnerabilities. To
> do DI safely, you have to validate your inputs and D's compile-time
> reflection provides a perfect way to implement validated DI. Every time
> I hear about some form of injection, be it SQL or JSON or Reflection,
> hit the news, makes me think that 5-10 years from now well look back on
> the use of unvalidated dynamic code constructs the same way we do about
> null terminated arrays today.

Considering security vulnerabilities is of course a very important 
matter. However, I think security must be assured a by software's 
design. Enforcing it solely by programming language constructs will not 
work for all cases. Particularly not, if this programming language 
allows direct access to a computer's memory, like C, C++, and D does. 
There is no sandbox out-of-the-box, where the compiled program runs in. 
So if you have a private field in a class that carries a password, one 
has to make sure it's only in memory as long as it is required. And if 
it is not required anymore one should clear it out with zeros, for 
example. So making a field private is by no means a secure solution. A 
hacker will not give up just because the official API declares a field 
to be private.

I consider 'unsafe reflection', as you call it, as a tool. Let's compare 
it to a knife, for instance. It is really a useful tool, I think nobody 
doubts, but at the same time it can be very dangerous. But inspite of 
this fact, everybody has not only one at home, I guess. Pointers are a 
useful tool, too. But they are also dangerous and still available in D, 
which is a good thing, because you can use this tool where needed. And 
actually, I don't think runtime reflection must be unsafe. The 
reflection mechanism provides type information for an object's 
properties, functions, arguments, etc. So validated DI is even possible 
with runtime reflection. Let's consider the the XML configuration of 
object trees once again. If you want to store a string to an int-field, 
for instance, then the DI container can refuse to do this, because it 
has access to all information required to enforce correct value types.

One more use case for reflection is data binding with GUI components. 
This approach is heavily used in Windows Presentation Foundation library 
[3,4]. GUI components can update an object's properties by using 
reflection. You don't need to register listeners for this purpose anymore.

That said, I don't think dynamic code constructs will be old-fashioned 
in 5-10 years, because you don't have to go an "unvalidated dynamic" way 
with runtime reflection.

BTW, as this thread also discusses an opt-in or opt-out implementation 
for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I 
guess the binaries are meant) is not as bad as it might sound with 
today's memory sizes. And if one wants to avoid it for optimization 
purpose, one can do it. The use of D as a systems programming language 
for embedded systems is, as I've read somewhere, not a first citizen 
anyway, because you'll get a lot of code for the garbage collection 
mechanism.

Cheers,
Jonny

[1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29
[2] http://en.wikipedia.org/wiki/Nhibernate
[3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation
[4] 
http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx