Compiler patch for runtime reflection

[Robert Jacques]

On Mon, 24 Oct 2011 12:38:12 -0400, Jacob Carlborg <doob at me.com> wrote:

> On 2011-10-24 17:37, Robert Jacques wrote:
>> I'm sorry if I've come across that way. I'm well aware with the reasons
>> for wanting access to private methods/fields via reflection and have
>> mentioned them in previous posts. What I've tried to point out is very
>> eloquently stated by fig. 6.3 on page 204 of TDPL: allowing reflection
>> to bypass protection guarantees essentially makes all declarations
>> public in nature, if not extern. That's a very heavy price to pay, just
>> from a program maintenance perspective. And if you consider someone
>> writing medical or financial software, the privacy concerns of exposing
>> private variable to all become very real.
>
> If someone is using a library and starts to mess with calling private
> methods they shouldn't they'll most likely mess things up for them self.

Aren't private methods by definition something people shouldn't be using?

>> Back to my actual argument, what I was pointing out was that the
>> toolboxes of various languages differ heavily. Just because Ruby has a
>> very cool library which uses runtime reflection, one shouldn't simply
>> state 'D must have runtime reflection.' One needs to first decompose
>> what the library is trying to achieve from the language mechanisms used
>> to implement it. Then you can ask yourself 1) Can I do that in D? 2) How
>> does D's solution compare to Ruby's?
>
> Of course. I'm just showing that another language/library uses it and
> therefore there could be a chance that should be doable in D as well.

I understand and appreciate your intent. However, do so should be accompanied by a critical dissection of the language/library, both to get those unfamiliar up to speed and to frame the conversation in terms of how best can we improve D.

>> My impression of what ActiveRecord, (which my be wrong), is that its
>> using runtime-reflection to do various meta-programming activities and
>> that under-the-hood, it stretching Ruby to the breaking point in order
>> to do so. D, on the other hand, has a ton of meta-programming specific
>> features, and doesn't need to rely on reflection to do so. This is
>> partly to do with Ruby supporting 'eval' and thus making no distinction
>> between compile-time and run-time behavior; that's correct for Ruby, but
>> D is a different kettle of fish.
>
> I doubt that ActiveRecord is stretching Ruby to the breaking point.

:) I don't know myself. I've just heard that the internals of Rails are deep, dark and full of non-idiomatic Rudy code.

> On
> the other hand it feels like I'm stretching D's metaprogramming
> capabilities in my serialization library. That's just because D doesn't
> have proper reflection.

I've written three serialization libraries and two versions of a runtime reflection library; the first one was prior to the string bugs in CTFE being fixed and ran DMD out of memory. I found serialization to be simple and straight forward. And I found compiler bugs to be a greater issue with reflection than any limits in D's capabilities. Indeed, given I wrote the same library using two completely different mechanisms, gives me great hope for D. So, I don't agree with your assessment, but I do understand where you're coming from. A lot of D's metaprogramming capabilities are undocumented, and knowing the right tool for the job is essential to getting nice, clean code.



> In Ruby there is very seldom reason to use "eval". That's because of the
> excellent support for runtime reflection. There's no clumsy syntax, no
> hacks, no need to instantiate reflectors and similar. You just call a
> method like you would call any other method:
>
> class Foo
>      def bar
>      end
> end
>
> foo = Foo.new
> foo.bar()
> foo.send("bar")
>
> In the above code, "bar" is called first using regular method call
> syntax and then using reflection.
>
>>>> I mean, considering that Ruby is a dynamic language, would bar even be
>>>> considered private from inside the super-class's ctor? Is Ruby's private
>>>> even comparable to D's private? Given that Ruby, by design, requires
>>>> fields to be private and for all external accesses to happen via
>>>> methods, should we consider a way to violate that contract an example of
>>>> mis-feature?
>>>
>>> There is no constructor involved here. "before_save" is class method
>>> called when the class "Foo" is loaded.
>>
>> So it's the static ctor, then.
>
> I guess you could call it that.

(Just checking, but you do know D has static ctors.)

>>> You can do basically whatever you want with the code in Ruby. You can
>>> replace an existing method/class in the standard library with your own.
>>> You can easily add methods to existing classes. It's the programmer that
>>> is responsible to not mess up things, just as in D where you can cast
>>> away immutable, shared and do unsafe things with pointers.
>>
>> Well one one hand, none of those things are allowed in SafeD (And you
>> know, that's not a resounding argument for reflection of private
>> members). On the other, you didn't answered my other two questions. In
>> fact, the impression I'm getting is that Ruby's private isn't comparable
>> to D's private.
>
> If we talk about methods it's easiest. If you call a private method in
> Ruby using the regular method call syntax you get an error. If you do
> the same in D you get an error as well. The only difference is in Ruby
> it will be a runtime error and in D it will be a compile time error.
>
> In Ruby you can bypass private using reflection with the "send" method.
> In D you can bypass private using a delegate.

Actually, you're not supposed to be able to; this is bug 1983. http://d.puremagic.com/issues/show_bug.cgi?id=1983  and  https://github.com/D-Programming-Language/dmd/pull/71

> In Ruby an instance variable is always private. I don't think you can
> declare a public instance variable, one uses setters/getters for that.
> In D you can declare both private and public instance variables.
>
> In Ruby you can get/set a value of an instance variable using
> reflection, instance_variable_set/get.
> In D you can get/set a value of a private instance variable using
> compile time reflection, i.e. .tupleof.

Yes, but only for source code you control.

> Now, since we can call private methods and get/set private instance
> variables using the features we have now, why shouldn't we be able to do
> the same with runtime reflection?
>
> BTW, I don't think it's a mis-feature.

Actually, baring bugs in DMD, .tupleof is the only way to access private fields and there is _no way_ to access private members; And .tupleof only works if you have access to the source code. Furthermore, .tupleof, god-send that it is, isn't part of the D spec; unlike .stringof, it didn't make it into TDPL nor into the D documentation. D's official method of compile-time reflection is __traits. And __traits doesn't bypass protection attributes. So precedence would be against runtime-reflection having access to private members.