Compiler patch for runtime reflection

[Robert Jacques]

On Tue, 25 Oct 2011 03:23:40 -0400, Jacob Carlborg <doob at me.com> wrote:
> On 2011-10-25 07:24, Robert Jacques wrote:
[snip]

>>> On
>>> the other hand it feels like I'm stretching D's metaprogramming
>>> capabilities in my serialization library. That's just because D doesn't
>>> have proper reflection.
>>
>> I've written three serialization libraries and two versions of a runtime
>> reflection library; the first one was prior to the string bugs in CTFE
>> being fixed and ran DMD out of memory. I found serialization to be
>> simple and straight forward. And I found compiler bugs to be a greater
>> issue with reflection than any limits in D's capabilities. Indeed, given
>> I wrote the same library using two completely different mechanisms,
>> gives me great hope for D. So, I don't agree with your assessment, but I
>> do understand where you're coming from. A lot of D's metaprogramming
>> capabilities are undocumented, and knowing the right tool for the job is
>> essential to getting nice, clean code.
>
> For example, getting the name of a instance variable:
>
> T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];
>
> I wouldn't consider that simple and straight forward. This works for
> both D1 and D2, don't know if D2 has a better way.

Not really, for the advanced user. I do find using "some string".length to be cleaner in these cases where you have to slice .stringof. i.e.

     static Value from(T)(T value) {
         //...
         static if(is(T == struct) || is(T == class)) {
             Value[string] result;
             foreach(i,v;value.tupleof) {
                 result[value.tupleof[i].stringof["value.".length..$]] = from(v);
             }
             return Value( result );
         }
         //..
     }

That said, I think anyone can read the above and understand what's happening. And even writing that code would be straightforward is .stringof and .tupleof were better documented.

Also, the first serialization library I wrote used __traits instead. It was more awkward, mainly because __traits doesn't an 'allFields' member, just an 'allMembers'. If it did then the above becomes:

foreach(name;__traits(allFields,value) {
     result[name] = from(__traits(getMember, value, name));
}

which really is straightforward. Now I originally used 'allMembers' and then filtered out everything I didn't want, which was fairly straightforward but verbose.

> Your serialization library only works on public fields? Doesn't sound
> very useful.

In the interest of full disclosure, the library I wrote was quick and dirty and does not handle ploy-morphism, cycle detection, aliasing, etc (mainly because JSON doesn't); I used traits for my first one (so protection was observed) but .tupleof for my second and third and was lazy about it. It's one line of code to add protection to tuple of, but I've never gotten around to it. I figured my code would be replace by orange or std.serialization or something.

That said, I have never serialized a private variable and have never wanted too. The serialized objects are, for the most part, part of the external API to my program. I have to do validation checks on the inputs and changing anything takes a week to get the other teams to propagate the changes on their side. There are also a couple of classes which I serialize in a binary format as a poor mans database, but again, there's no need for anything in it to be private.

[snip]

>> Actually, you're not supposed to be able to; this is bug 1983.
>> http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
>> https://github.com/D-Programming-Language/dmd/pull/71
>
> I don't see what that has to do with private, it's about const, as far
> as I can see.

Both are protection attributes. The problem of taking the delegate of a mutable member on a const object, is the exact same problem as taking the delegate of a private member.

In short, you can't 'bypass private using a delegate.' That you can today, is simple a bug in the compiler.

> I don't see how calling a private method through a
> delegate can be easily solved. I see three solutions:
>
> 1. Make it a compile time error to create a delegate from a private
> method. This would not allow to call that method via a delegate in the
> same module, which is possible if you call the method directly.
>
> 2. Implement some, most likely, very advanced data flow analysis in the
> compiler to detect when a delegate pointing to a private method escapes
> the module. Results in a compile time error. Will most likely not happen.
>
> 3. Make it a runtime error when the delegate is actually called. This
> would probably need some pieces of 2 as well. I don't think the
> community will be happy about adding runtime checks like these.
>
> Oh, BTW the same needs to be done for function pointers as well since
> you can build a delegate out of a function pointer and an object.

Bypassing the protection type system and delegate escape are very different things. The escape of a delegate is fully controlled by the coder of the class/module; with bypass mechanisms, the library writer has no control.

>>> In Ruby an instance variable is always private. I don't think you can
>>> declare a public instance variable, one uses setters/getters for that.
>>> In D you can declare both private and public instance variables.
>>>
>>> In Ruby you can get/set a value of an instance variable using
>>> reflection, instance_variable_set/get.
>>> In D you can get/set a value of a private instance variable using
>>> compile time reflection, i.e. .tupleof.
>>
>> Yes, but only for source code you control.
>
> You don't need control of the source, you only need the static type. Of
> course an opaque struct won't work.

Sorry, that's what I meant by source you control. i.e. it is available to you to read in some form. Runtime reflection would allow you access to knowledge of opaque types.

>> Actually, baring bugs in DMD, .tupleof is the only way to access private
>> fields and there is _no way_ to access private members; And .tupleof
>> only works if you have access to the source code. Furthermore, .tupleof,
>> god-send that it is, isn't part of the D spec; unlike .stringof, it
>> didn't make it into TDPL nor into the D documentation. D's official
>> method of compile-time reflection is __traits. And __traits doesn't
>> bypass protection attributes. So precedence would be against
>> runtime-reflection having access to private members.
>
> The compiler has .tupleof and it's mentioned in the spec at
> d-programming-language.org: http://d-programming-language.org/struct.html

I stand corrected. Sorry, I just looked at TDPL's index, the page on Properties and googled it. Now that I look for it, it's also on the Class page.

Still, __traits is still the full fledged compile-time reflection feature and doesn't bypass protection attributes.