Compiler patch for runtime reflection

[Adam Wilson]

On Wed, 26 Oct 2011 11:20:47 -0700, Jonny Dee <jonnyd at gmx.net> wrote:

> Hello Robert,
>
> Am 26.10.11 07:16, schrieb Robert Jacques:
>> On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd at gmx.net> wrote:
>>> Am 25.10.11 16:41, schrieb Robert Jacques:
>>>> On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd at gmx.net> wrote:
>>>>> [...]
>>> Hi Robert,
>>>
>>> Well, before I tell you what I would like to see I'll cite Wikipedia  
>>> [1]:
>>> "
>>> [...]
>>> - Discover and modify source code constructions (such as code blocks,
>>> classes, methods, protocols, etc.) as a first-class object at runtime.
>>> - Convert a string matching the symbolic name of a class or function
>>> into a reference to or invocation of that class or function.
>>> [...]
>>> "
>>>
>>> Here is what I would dream of for arbitrary objects/classes (not
>>> necessarily known at compile-time):
>>> - Query an object for its list of methods together with their
>>> signatures. Select a method, bind some values to its arguments, call  
>>> it,
>>> and retrieve the return type (if any).
>>> - Query an object for its public fields (at least), and provide a way  
>>> to
>>> get/set their values.
>>> - Query an object's class for all implemented interfaces and its base
>>> class.
>>> - Query a module for all type definitions and provide a way to
>>> introspect these types in more detail. For instance, it would be really
>>> cool if I could find a class with name "Car" in module "cars", get a
>>> list of all defined constructors, select one, bind values to the
>>> constructor's parameters, and create a corresponding object.
>>>
>>> [...]
>>>
>>> Implementing such a DI container heavily depends on reflection, because
>>> the DI container component doesn't know anything about the objects to  
>>> be
>>> created during runtime.
>>>
>>> Qt also extends C++ with a reflection mechanism through the help of its
>>> meta object compiler (moc). It analyses the C++ source code, generates
>>> meta class definitions [6,7] and weaves them into your Qt class. Hence,
>>> in Qt, you can query an object for fields, methods, interfaces, etc.  
>>> and
>>> you can call methods with arbitrary parameters, or you can instantiate  
>>> a
>>> class using an arbitrary constructor. Consequently, somone implemented  
>>> a
>>> DI container for C++ which is based on Qt and works more or less the
>>> same way the Spring DI container does. You can build up object trees
>>> simply by specifying such trees in an XML file.
>>>
>>> I don't go into why dependency injection is a very powerful feature.
>>> This is Martin Fowler's [3] job ;) But when I program with C++ I miss
>>> such a flexible dependency injection mechanism a lot. And I hope this
>>> will eventually be available for D.
>>>
>>> Cheers,
>>> Jonny
>>>
>>> [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
>>> [2] http://en.wikipedia.org/wiki/Dependency_injection
>>> [3] http://martinfowler.com/articles/injection.html
>>> [4]
>>> http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
>>>
>>> [5] http://qtioccontainer.sourceforge.net/
>>> [6] http://doc.qt.nokia.com/stable/qmetaobject.html
>>> [7]
>>> http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx
>>>
>>
>> Hi Jonny,
>> Thank you for your informative (and well cited) post. It has provided me
>> with a new take on an old design pattern and some enjoyable reading. In
>> return, let me outline my opinion of reflection in D today, and
>> tomorrow, as it pertains to your wish list.
>
> Many thanks to you, too, for your very elaborate answer :)
>
>> Reflection in D today is very different from the host of VM languages
>> that have popularized the concept. Being a compiled systems language,
>> actual runtime self-modification is too virus like to become at a
>> language level feature. However, given the compilation speed of D,
>> people have made proof of concept libraries that essentially wrapped the
>> compiler and dynamically loaded the result. As LDC uses LLVM, which has
>> a jit backend, I'd expect to see something get into and D 'eval' library
>> into etc eventually. (phobos uses the BOOST license, which isn't
>> compatible with LLVM).
>
> I know, that "runtime self-modification" and runtime code generation is  
> a "dangerous" feature. And there really are rare cases where using such  
> an approach might justify the risc in using it. Although this feature is  
> not on my wish list, it might be good for generating dynamic proxies to  
> arbitrary object instances like they are used by some ORMs. See  
> Hibernate/NHibernate, for example [1,2]. Another example is  
> aspect-oriented programming. But while I can't see the exacty reason for  
> it, such a feature might indeed be a feature which is more appropriate  
> for VM languages.
>
>> Compile-time reflection and generation of code, on the other hand, is
>> something D does in spades. It fulfills your dream list, although I
>> think module level reflection might only be available in the github
>> version. The API design is still in flux and we are actively iterating /
>> improving it as find new uses cases and bugs. The current plan is to
>> migrate all the traits functions over to a special 'meta' namespace
>> (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
>> libraries for each of the concepts I listed, (prototype objects,
>> duck-typing/casting or serialization), have been written using the
>> compile-time meta-programming features in D. So that's the good.
>
> A absolutely agree! D's compile-time reflection is very good. And it's  
> hard to top this.
>
>> On the other hand, D's runtime capabilities are limited to
>> object.factory, the under implemented RTTI and library solutions which
>> manually expose information gathered by D's compile-time mechanisms. And
>> so far, these tools have been more than enough, from a functionality
>> point of view. Most of our desire for better runtime reflection stems
>> from a desire for efficiency, composition, cleanliness of user syntax
>> and simplification of library code. These are all important issues for
>> the widespread use of reflection based libraries, but they're not 'I
>> can't implement X' issues.
>
> You are right, there IS certainly always some way to 'implement X'. But,  
> as you know, there is always a consideration of the effort you need to  
> implement X.
>
>> As for the future, I believe that the division in D between compile-time
>> and run-time reflection warrants a serious look at the design of the
>> run-time half of the system. To that end, I believe that implementing
>> reflection in a library would be the best way to experiment and iterate
>> an API. To that end, I have a proposal in the review queue to improve
>> std.variant which contains dynamic dispatch (i.e. the ability to get/set
>> public fields and call methods), duck-typeing/casting and
>> prototype-style objects. Notably, it's missing introspection
>> capabilities as thats what I'm most unsure about API wise, and simplest
>> to add. Designing reflection inside a library keeps Walter & Co's
>> bandwidth free for other bugs/features and provides a very good stress
>> test of D's meta-programming capabilities. (Both of which I think are
>> good things)
>
> I've got no problem with an approach which puts runtime reflection  
> capabiities into a separate library. No matter were you look at, Java,  
> C#, or Qt, all have a library for reflection purpose. I do not see,  
> however, how this might be done without compiler support. As already  
> mentioned, Qt has its own moc compiler, which parses your C++ source  
> code and generates the necessary infra structure. I'm still a beginner  
> with respect to D, so I don't know what is really already possible, and  
> what not. But my current feeling is that a similar approach would also  
> be needed for D. If the D compiler itself, or another post-compiler  
> component should generate the meta information for runtime reflection is  
> another question. I could live with both, although I'd prefer the former  
> built-in one.
>
>> (I am soliciting feedback, if you care to take a look:
>> https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)
>
> Thanks for this link, I'll certainly have a look at it.
>
>> And there are many reasons we might want to experiment with D's runtime
>> reflection API instead of just copying someone. For example, take
>> Dependency Injection. Using Java-style reflection, DI suffers from the
>> Reflection injection and Unsafe Reflection security vulnerabilities. To
>> do DI safely, you have to validate your inputs and D's compile-time
>> reflection provides a perfect way to implement validated DI. Every time
>> I hear about some form of injection, be it SQL or JSON or Reflection,
>> hit the news, makes me think that 5-10 years from now well look back on
>> the use of unvalidated dynamic code constructs the same way we do about
>> null terminated arrays today.
>
> Considering security vulnerabilities is of course a very important  
> matter. However, I think security must be assured a by software's  
> design. Enforcing it solely by programming language constructs will not  
> work for all cases. Particularly not, if this programming language  
> allows direct access to a computer's memory, like C, C++, and D does.  
> There is no sandbox out-of-the-box, where the compiled program runs in.  
> So if you have a private field in a class that carries a password, one  
> has to make sure it's only in memory as long as it is required. And if  
> it is not required anymore one should clear it out with zeros, for  
> example. So making a field private is by no means a secure solution. A  
> hacker will not give up just because the official API declares a field  
> to be private.
>
> I consider 'unsafe reflection', as you call it, as a tool. Let's compare  
> it to a knife, for instance. It is really a useful tool, I think nobody  
> doubts, but at the same time it can be very dangerous. But inspite of  
> this fact, everybody has not only one at home, I guess. Pointers are a  
> useful tool, too. But they are also dangerous and still available in D,  
> which is a good thing, because you can use this tool where needed. And  
> actually, I don't think runtime reflection must be unsafe. The  
> reflection mechanism provides type information for an object's  
> properties, functions, arguments, etc. So validated DI is even possible  
> with runtime reflection. Let's consider the the XML configuration of  
> object trees once again. If you want to store a string to an int-field,  
> for instance, then the DI container can refuse to do this, because it  
> has access to all information required to enforce correct value types.
>
> One more use case for reflection is data binding with GUI components.  
> This approach is heavily used in Windows Presentation Foundation library  
> [3,4]. GUI components can update an object's properties by using  
> reflection. You don't need to register listeners for this purpose  
> anymore.
>
> That said, I don't think dynamic code constructs will be old-fashioned  
> in 5-10 years, because you don't have to go an "unvalidated dynamic" way  
> with runtime reflection.
>
> BTW, as this thread also discusses an opt-in or opt-out implementation  
> for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I  
> guess the binaries are meant) is not as bad as it might sound with  
> today's memory sizes. And if one wants to avoid it for optimization  
> purpose, one can do it. The use of D as a systems programming language  
> for embedded systems is, as I've read somewhere, not a first citizen  
> anyway, because you'll get a lot of code for the garbage collection  
> mechanism.
>
> Cheers,
> Jonny
>
> [1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29
> [2] http://en.wikipedia.org/wiki/Nhibernate
> [3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation
> [4]  
> http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx

I am a first time poster but I've been lurking around here for a while now  
and this topic is something that is extremely near and dear to my heart. I  
have been working with .NET since the C# 1.0 Beta 1 days and cut my teeth  
on WPF while it was still called Avalon. I know you mention that WPF uses  
Reflection, and on that statement I can only say that you are correct, for  
better or worse, usually worse. However, thats not the complete story. WPF  
uses something called a DependencyObject to implement objects with  
data-binding. And this is important as literally EVERY WPF object inherits  
this class, you can't build a WPF object without it. And every property  
you set on a WPF object is really two properties, one of which is a  
"public static DependencyProperty PropertyName", a DependencyProperty is  
also being a class that requires a DependencyObject to initialize it. In  
WPF you absolutely MUST declare this object as public. And while not  
strictly required Microsoft Best Practices dictate that the actual  
property is public as well.

My point is that even in WPF you don't actually NEED reflection of private  
members. The reason for this is that back during the design of Silverlight  
Microsoft knew that SL's reflection would NOT be able to reflect to  
private members and they needed to make sure that SL XAML would still work  
with WPF XAML. Remember, SL was originally called WPF/Everywhere. WPF has  
taken a lot of flack for the amount of reflection that it does precisely  
because it slows it down, but I think that it has other bigger problems.  
Personally, I've found the Measure and Arrange calls to be even  
significantly more expensive, but those can be mitigated with experience,  
the reflection can't, so it gets blamed. WPF has many other perf issues  
too, how they draw rounded corners makes grown men cry and any game  
programmer who wrote code that bad would be fired on the spot, a one  
thousand polygon circle anyone?

It's also interesting that you bring up WPF because the project I lead,  
The Horizon Project, is an attempt to create a open-source multi-platform  
framework that is as easy to use as the .NET Framework and one of it's  
core pillars is a declarative XML based UI design language. I am  
evaluating D and so far it is blowing C++ out of the water, (we'll need to  
have long conversations about strings though, I think .NET has the right  
answer here) but I have to admit that I am seriously missing Reflection.  
The problem is that there is currently no way to do data-binding without  
member-level reflection because it is a purely run-time concept. That  
said, I am researching ways around this, and I do have ideas ... but  
that's not for this forum.

For D, I have two notes. First, data-binding is the way of the future, you  
can't ignore it, or say that because it doesn't fit with philosophy 'X'  
that it shouldn't be done. The biggest reason that open-source projects  
die is that the industry moves in a way that conflicts with the projects  
philosophy and leaves the project behind. And this is a good thing, we are  
collective discovering better ways of doing things and part of that  
process is rejecting old philosophies. Adapt or die. The second is that  
you can't protect the programmer from all possible mistakes, and in terms  
of security their are many worse potential exploits that are much easier  
to reach for than Reflection. And to be honest, I've never actually heard  
of a successful Reflection exploit via the .NET Framework, although I am  
sure someone here knows of one...

However, I see no particular reason to reflect on private members by  
default, if you need to get data out, you should be using a public  
property or method to filter the data as needed, that's kind of the point  
of scoping. Private member reflection is NOT a requirement for UX, WPF  
being the case in point. I personally like the solution that Reflection in  
D is opt-out with both command-line options and @noreflect. I would add  
that a @deepreflect option be made available for those incorrigible souls  
who absolutely must reflect private members. I would also add that  
reflection should be default disabled by the compiler for code the works  
directly with pointers and direct memory access (malloc/free, new/delete,  
etc.) and the developer can only enable it using a compiler option.

For most of us, code-bloat is not an issue even in the slightest, todays  
computers can open multi-megabyte executables running JIT's (think .NET)  
in less than half a second (we have a working example where I work). In  
this case, a little extra bloat to make my life significantly easier is a  
non-choice, I'll take the bloat every time. However, I completely agree  
that there are cases where bloat is bad, anything system level (kernels,  
drivers, etc.), mobile or power-limited devices (although more of a  
toss-up), and others like games. But in the end, I have products to  
deliver, I need to best tools for the job, if that means using reflection,  
so be it, ideology should never enter the equation. In the case of  
reflection, it is my opinion that the compiler should provide a full range  
of tools and let the developers decide how much they want to use; this  
would make D superior to even .NET, as .NET is extremely limited in it's  
reflection control.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.com/