Compiler patch for runtime reflection

[Jacob Carlborg]

On 2011-10-25 16:21, Robert Jacques wrote:
> On Tue, 25 Oct 2011 03:23:40 -0400, Jacob Carlborg <doob at me.com> wrote:
>> For example, getting the name of a instance variable:
>>
>> T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];
>>
>> I wouldn't consider that simple and straight forward. This works for
>> both D1 and D2, don't know if D2 has a better way.
>
> Not really, for the advanced user. I do find using "some string".length
> to be cleaner in these cases where you have to slice .stringof. i.e.
>
> static Value from(T)(T value) {
> //...
> static if(is(T == struct) || is(T == class)) {
> Value[string] result;
> foreach(i,v;value.tupleof) {
> result[value.tupleof[i].stringof["value.".length..$]] = from(v);
> }
> return Value( result );
> }
> //..
> }
>
> That said, I think anyone can read the above and understand what's
> happening. And even writing that code would be straightforward is
> .stringof and .tupleof were better documented.

Yes, I found the snippet I use to get the name of a field by trail and 
error.

> Also, the first serialization library I wrote used __traits instead. It
> was more awkward, mainly because __traits doesn't an 'allFields' member,
> just an 'allMembers'. If it did then the above becomes:
>
> foreach(name;__traits(allFields,value) {
> result[name] = from(__traits(getMember, value, name));
> }
>
> which really is straightforward. Now I originally used 'allMembers' and
> then filtered out everything I didn't want, which was fairly
> straightforward but verbose.

__traits(getMember) is more clean when you the name of all the fields 
you want.

>> Your serialization library only works on public fields? Doesn't sound
>> very useful.
>
> In the interest of full disclosure, the library I wrote was quick and
> dirty and does not handle ploy-morphism, cycle detection, aliasing, etc
> (mainly because JSON doesn't); I used traits for my first one (so
> protection was observed) but .tupleof for my second and third and was
> lazy about it. It's one line of code to add protection to tuple of, but
> I've never gotten around to it. I figured my code would be replace by
> orange or std.serialization or something.

Well, things get a bit more complicated when the library needs to 
support ploy-morphism, cycle detection, aliasing and so on.

> That said, I have never serialized a private variable and have never
> wanted too. The serialized objects are, for the most part, part of the
> external API to my program. I have to do validation checks on the inputs
> and changing anything takes a week to get the other teams to propagate
> the changes on their side. There are also a couple of classes which I
> serialize in a binary format as a poor mans database, but again, there's
> no need for anything in it to be private.

Ok, I see.

>
> [snip]
>
>>> Actually, you're not supposed to be able to; this is bug 1983.
>>> http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
>>> https://github.com/D-Programming-Language/dmd/pull/71
>>
>> I don't see what that has to do with private, it's about const, as far
>> as I can see.
>
> Both are protection attributes. The problem of taking the delegate of a
> mutable member on a const object, is the exact same problem as taking
> the delegate of a private member.

I don't agree.

> In short, you can't 'bypass private using a delegate.' That you can
> today, is simple a bug in the compiler.
>
>> I don't see how calling a private method through a
>> delegate can be easily solved. I see three solutions:
>>
>> 1. Make it a compile time error to create a delegate from a private
>> method. This would not allow to call that method via a delegate in the
>> same module, which is possible if you call the method directly.
>>
>> 2. Implement some, most likely, very advanced data flow analysis in the
>> compiler to detect when a delegate pointing to a private method escapes
>> the module. Results in a compile time error. Will most likely not happen.
>>
>> 3. Make it a runtime error when the delegate is actually called. This
>> would probably need some pieces of 2 as well. I don't think the
>> community will be happy about adding runtime checks like these.
>>
>> Oh, BTW the same needs to be done for function pointers as well since
>> you can build a delegate out of a function pointer and an object.
>
> Bypassing the protection type system and delegate escape are very
> different things. The escape of a delegate is fully controlled by the
> coder of the class/module; with bypass mechanisms, the library writer
> has no control.

So how should this be fixed? Make it a compile error to create a 
delegate from a private method?

>>>> In Ruby an instance variable is always private. I don't think you can
>>>> declare a public instance variable, one uses setters/getters for that.
>>>> In D you can declare both private and public instance variables.
>>>>
>>>> In Ruby you can get/set a value of an instance variable using
>>>> reflection, instance_variable_set/get.
>>>> In D you can get/set a value of a private instance variable using
>>>> compile time reflection, i.e. .tupleof.
>>>
>>> Yes, but only for source code you control.
>>
>> You don't need control of the source, you only need the static type. Of
>> course an opaque struct won't work.
>
> Sorry, that's what I meant by source you control. i.e. it is available
> to you to read in some form. Runtime reflection would allow you access
> to knowledge of opaque types.

Ok, actually I haven't thought of how runtime reflection would be have 
with an opaque type.

>>> Actually, baring bugs in DMD, .tupleof is the only way to access private
>>> fields and there is _no way_ to access private members; And .tupleof
>>> only works if you have access to the source code. Furthermore, .tupleof,
>>> god-send that it is, isn't part of the D spec; unlike .stringof, it
>>> didn't make it into TDPL nor into the D documentation. D's official
>>> method of compile-time reflection is __traits. And __traits doesn't
>>> bypass protection attributes. So precedence would be against
>>> runtime-reflection having access to private members.
>>
>> The compiler has .tupleof and it's mentioned in the spec at
>> d-programming-language.org: http://d-programming-language.org/struct.html
>
> I stand corrected. Sorry, I just looked at TDPL's index, the page on
> Properties and googled it. Now that I look for it, it's also on the
> Class page.

No problem.

> Still, __traits is still the full fledged compile-time reflection
> feature and doesn't bypass protection attributes.

Yeah, but for a serialization library I want to be able to serialize 
private fields. This is known issue that this breaks encapsulation.

-- 
/Jacob Carlborg