I give up! I tried to create a reflection library but D's traits are just too screwed up!

[Alex]

On Wednesday, 3 April 2019 at 17:56:08 UTC, Andrei Alexandrescu 
wrote:
> On 4/3/19 3:22 AM, Alex wrote:
>> The follow code is a reflection library that unifies all the 
>> junk in std.traits and __traits and makes it using reflection 
>> as if it were runtime and one can even keep the info around at 
>> runtime...
>> 
>> But D's type system is too screwed up.
>
> Alex, thank you for this work. After looking at it, I was all 
> morning on the phone with my students Eduard Staniloiu and 
> Razvan Nitu, discussing in good part how to make inroads into 
> the introspection matter.
>
> Introspection is important. It is also, more or less 
> incidentally, central to Eduard's doctoral dissertation and 
> also important for Razvan's.
>
> They will look through your code trying to identify and look at 
> fixes for the problems you encountered. One thing you could 
> help with is to break down the problems you found in bite-size 
> chunks - a simple example, the expected behavior, and the 
> actual behavior.
>
> In some cases it may be something desirable but not a 
> showstopper. Example: "D forces code duplication". In some 
> other cases there may be no code example, e.g. "I need 
> introspection artifact X to do Y and there isn't any."
>
> Please assign all of these issues to Eduard.
>
>
> Thanks!

Thanks for taking this serious enough to take it serious enough! 
(meaning looking in to it).

1. One of the problems here is that as I wrote the code and came 
up against problems I'd have to try something and if it didn't 
work I'd move on to try something else not taking in to account 
why it specifically didn't work.

I will try to break it down though in to pieces.

First, the code is relatively straight forward:

Oop is used because I thought it would better unify CT and RT.

The core feature is this:

abstract class cBaseReflection
{
	// Declared in reverse order for proper output
	sAttributeReflection[] Attributes;			// Attributes on the type
	cBaseReflection[] Uses;						// Known types that use this type 
as a field, parameter, return type, etc
	string Body;								// D does not allow getting a body but this 
is included if it ever does
	string Protection;
	string MangledName;
	string ModuleName;							// The module name
	string FullName;							// The fully qualified name
	string TypeName;							// The type name
	string Id;									// The id  = __traits(identifier,T)
	
	string DType() { return "Unknown"; }

	auto Reflect(alias T)()
	{
		
		Id = __traits(identifier, T);	
		static if (__traits(compiles, T.stringof)) TypeName = 
T.stringof;
		static if (__traits(compiles, moduleName!T)) ModuleName = 
moduleName!(T);
		static if (__traits(compiles, fullyQualifiedName!T)) FullName = 
fullyQualifiedName!(T);
		static if (__traits(compiles, mangledName!T)) MangledName = 
mangledName!T;
		static if (__traits(compiles, __traits(getProtection, T))) 
Protection = __traits(getProtection, T);
		// Get the attributes for the type
		static if (__traits(compiles, __traits(getAttributes, T)))
			static foreach(a;  __traits(getAttributes, T)) Attributes ~= 
sAttributeReflection(to!string(a), typeof(a).stringof);

		return this;
	}
}



cBaseReflection is the prototype for all the classes. It simply 
contains the common type info of all types. All the other classes 
are just specializing for the different D types but have the same 
structure.

Reflect is a common templated method who's purpose is to fill out 
that information using __traits and std.traits. These then wrap 
those functions so we don't have to call them by hand, we just 
Reflect!T and it will return the class with the data filled out.



For example, a typical problem in D meta programming is to get 
the enum members names and values, the following specialization 
of cBaseReflection does this:

/*
Encapsulates an Enumeration Reflection
*/
class cEnumReflection : cBaseReflection
{
	static struct sValueReflection
	{
		string Name;
		string Value;		
		sAttributeReflection[] Attributes;			
		this(string n, string v) { Name = n; Value = v; }
	} sValueReflection[] Values;

	string BaseType;
	override string DType() { return "enum"; }

	auto Reflect(alias T)()
	{
		super.Reflect!T;
		
		//BaseType = TypeOf(T).stringof;
		static foreach(e; EnumMembers!T)
		{{
			Values ~= sValueReflection(to!string(e), e.stringof);
			mixin("alias E = __traits(getAttributes, 
T."~to!string(e)~");");			
			static foreach(a; E) 						
				Values[$-1].Attributes ~= sAttributeReflection(to!string(a), 
typeof(a).stringof);							
		}}

		return this;
	}
}


Normally we would have to manually do the for each in our code, 
now we can just Reflect!SomeEnum.Values[k].Value to get the kth 
value.

This is much easier than having to remember the specifics of the 
`static foreach(e; EnumMembers!T)` because sometimes there are 
__traits to use and sometimes their are templates. Sometimes one 
has to use typeof and other times TypeOf(becausse of aliasing 
issues).

So Reflect! Abstracts all those issues away(ideally).

Reflect!T calls the parent base Reflect to fill in the more 
general info: super.Reflect!T;

The idea is simple, to reduce duplicate code. There should be no 
reason to have to duplicate code to get the same info. Since 
cBaseReflection.Reflect gets all the general info(and we can use 
__traits(compiles,) to handle any minor discrepancies, let 
cBaseReflection.Reflect do it's job for all specialized types and 
it puts the code in one singular place.


All this basically works without a hitch(although I have 
duplicated getting the attributes for the enum. We could possibly 
let cBaseReflection or add a Reflect!T to sAttributeReflection to 
handle it.







Looking at cAggregateReflection, we have


/*
	Encapsulates an Aggregate Type
*/
abstract class cAggregateReflection : cBaseReflection
{		

	
	
	cMethodReflection[] Methods;							// The methods
	cFieldReflection[] Fields;								// The fields
	cClassReflection[] DerivedClasses;						// Known immedate 
derived class that derive this type
	cAggregateReflection[] NestedAggregates;				// Any nested 
aggregates
	sTypeReflection[] AliasThis;							// The Alias This	
	sTypeReflection[] TypeParameters;						// Any type parameters 
used.
	bool HasAliasing;
	bool HasElaborateAssign;
	bool HasElaborateCopyConstructor;
	bool HasElaborateDestructor;
	bool HasIndirections;
	bool HasNested;
	bool HasUnsharedAliasing;
	bool IsInnerClass;
	bool IsNested;
	
	override string DType() { return "Unknown"; }

	auto Reflect(alias T)()
	{
		super.Reflect!T;

		// Get the Alias This
		static foreach(a;  __traits(getAliasThis, T))
		{{
			mixin("import "~moduleName!T~";");		
			mixin("AliasThis ~= sTypeReflection(to!string(a), 
typeof("~moduleName!T~"."~T.stringof~"."~to!string(a)~").stringof);");
		}}

		HasAliasing = hasAliasing!T;
		HasElaborateAssign = hasElaborateAssign!T;
		HasElaborateCopyConstructor = hasElaborateCopyConstructor!T;
		HasElaborateDestructor = hasElaborateDestructor!T;
		HasIndirections = hasIndirections!T;
		HasNested = hasNested!T;
		HasUnsharedAliasing = hasUnsharedAliasing!T;
		static if (__traits(compiles, isInnerClass!T)) IsInnerClass = 
isInnerClass!T;
		static if (__traits(compiles, isNested!T)) IsNested = 
isNested!T;
		
	
		// Get the fields
		alias n = FieldNameTuple!T;
		static foreach(k, f; std.traits.Fields!T)
		{{
			Fields ~= (new cFieldReflection()).Reflect!(Alias!T, n[k]);
		}}


		static foreach(k, m; __traits(derivedMembers, T))
		{{
			static if (__traits(compiles, __traits(getVirtualMethods, T, 
m)))
			static foreach(j, v; typeof(__traits(getVirtualMethods, T, 
m)))		
			{{
				//pragma(msg, T.stringof, " --- ", v.stringof);
				Methods ~= (new cMethodReflection()).Reflect!(T, m);

			}}
		}}
		return this;
	}
}


and one can see it's pretty straight forward: It's just wrapping 
all the basic aggreigate std.trait calls so we don't have to call 
them individually.

HasAliasing = hasAliasing!T;
HasElaborateAssign = hasElaborateAssign!T;
HasElaborateCopyConstructor = hasElaborateCopyConstructor!T;
HasElaborateDestructor = hasElaborateDestructor!T;
HasIndirections = hasIndirections!T;
HasNested = hasNested!T;
HasUnsharedAliasing = hasUnsharedAliasing!T;

It obviously calls the super.Reflect! so to avoid duplicating 
code.

An aggregate may not have methods or certain other entries but we 
include them in this class to be able to handle them more 
generally as to avoid duplicate code. Because we have 
__traits(compiles,) we can always ignore the cases that will fail 
very easily.

The code is quite simple. It's just really basic oop and wrapping 
a lot of the traits code, again, to abstract them away.

The general model is

class Child : Parent
{
    data
    auto Reflect(T)() { super.Reflect!T; ..fill out data.. return 
this; }
}

And calling child.Reflect will cascade all the calls and have 
each class in the hierarchy fill out the appropriate information 
for the type it is acting on(ideally there will be a 
corresponding D type(class, field, enum, interface, template, 
etc)).

If none of this helps in understanding the code you can reach me 
at Incipient at email.com and I'll try to explain it better.

The code was in it's initial stages as it was more prototyping 
and seeing what could be done. Ideally one would be able to get 
any piece of CT information one desires from Reflect!. I didn't 
get finished filling out all the details before I ran in to the 
issues below.


-------------------------------------------------
-------------------------------------------------


Now, when we get to filling out field info the issues start to 
occur. An aggregate may have fields and so we want to get the 
meta info for them:

	// Get the fields
	alias n = FieldNameTuple!T;
	static foreach(k, f; std.traits.Fields!T)
	{{
		Fields ~= (new cFieldReflection()).Reflect!(Alias!T, n[k]);
	}}


To get information about fields one has to use FieldNameTyple to 
get the string names of the Id's for the fields and 
std.traits.Fields!T to get the types.

//std.traits.Fields
template Fields(T)
{
     static if (is(T == struct) || is(T == union))
         alias Fields = typeof(T.tupleof[0 .. $ - isNested!T]);
     else static if (is(T == class))
         alias Fields = typeof(T.tupleof);
     else
         alias Fields = AliasSeq!T;
}


The problem here is that there is no Field Type in D. That is, a 
field is not a type in and of itself. A class is a type. So we 
have to pass the type and name that is returned, rather than a 
type which would encapsulate that info(and which could then be 
gotten inside cFieldsReflection and this would help encapsulate 
code)

 From the help:
import std.meta : AliasSeq;
struct S { int x; float y; }
static assert(is(Fields!S == AliasSeq!(int, float)));

So, if we want cFieldsReflection to do all the work we have to 
then not just parse the field type passed(since we can't) but 
pass the info that lets us build the field type.

This alone breaks the design of the code. Specifics are having to 
be handled in more general code, which reduces isolation:

E.g, ideally we could do this:

	// Get the fields TYPES
	static foreach(k, f; std.traits.FieldsTYPE!T)
	{{
		Fields ~= (new cFieldReflection()).Reflect!(f);
	}}

// TYPE for emphasis rather than Type

and then f, as a type(a type in the sense of a self contained 
compiler meta unit that can be passed around as a type. It may 
not make sense that it is a true D type but for meta programming 
it acts like one. E.g. hypothetically we could do

class X { int somefield; }
class Y { typeof(FieldsTYPE!(X)[0]) anotherintfield; }
).


So, instead we have to pass the best info we can that will let us 
hopefully get at the info we need to fill out the field data.

Fields ~= (new cFieldReflection()).Reflect!(Alias!T, n[k]);

Here T is passed(the Alias is left over from trying various work 
arounds), which is the class the field is in, and n[k] is the 
name of the kth field.  The idea is that we can build `T.(n[k])` 
such as someclass.somefield.


So this gets passed to

/*
Encapsulates a Field Reflection
*/
class cFieldReflection : cBaseReflection
{
	override string DType() { return "field"; }

	auto Reflect(alias T, string name)()
	{		
		Id = name;	
		static if (__traits(compiles, TypeName = T.stringof)) TypeName 
= T.stringof~"."~name;
		ModuleName = moduleName!(T);
		FullName = fullyQualifiedName!(T)~"."~name;
		MangledName = mangledName!T;
		//mixin(`FullName = mangledName!(`~T.stringof~`.`~name~`);`);
			

		// Get the attributes for the field
		import mModel;
		static if (__traits(compiles, mixin(`static if 
(__traits(compiles, __traits(getAttributes, 
`~T.stringof~`.`~name~`)))`)))
		mixin(`static if (__traits(compiles, __traits(getAttributes, 
`~T.stringof~`.`~name~`))) pragma(msg, "fdadsf"); `);
		//static if (aFound) { mixin(`static foreach(a;  
__traits(getAttributes, `~T.stringof~`.`~name~`)) Attributes ~= 
sAttributeReflection(to!string(a), typeof(a).stringof);`);}

		//pragma(msg, T);
		//T t;
		//mixin("alias X = t."~name~";");
		//super.Reflect!X;

		return this;
	}
}

Note that cBaseReflection.Reflect code is duplicated rather than 
just calling super.Reflect. Why?

Because we have no type to pass, and Reflect takes a type and 
extracts the info. We have to build the info manually using hacks 
such as:

Id = name;	// easy because the name is the id
TypeName = T.stringof~"."~name; // this just uses 
someclass.somefield

The rest just use T, the class or uses someclass.somefield and 
hopes for the best

ModuleName = moduleName!(T);
FullName = fullyQualifiedName!(T)~"."~name;
MangledName = mangledName!T;
//mixin(`FullName = mangledName!(`~T.stringof~`.`~name~`);`); // 
leftover from testing, was just using fullname for holding the 
string

For example, their is no mangling for the field(should their be, 
ultimately?). We just use the mangled name of the class, should 
work good enough but it is not general.



Now we have to get the attributes, again, cBaseReflection does 
all that but we can't call it because Reflect requires a type and 
cBaseReflection shouldn't try to handle fields using field names 
and the class.

// Get the attributes for the field
import mModel; // We have to import the model because we have no 
type that encapsulates it all.

static if (__traits(compiles, mixin(`static if 
(__traits(compiles, __traits(getAttributes, 
`~T.stringof~`.`~name~`)))`)))
mixin(`static if (__traits(compiles, __traits(getAttributes, 
`~T.stringof~`.`~name~`))) pragma(msg, "fdadsf"); `);
//static if (aFound) { mixin(`static foreach(a;  
__traits(getAttributes, `~T.stringof~`.`~name~`)) Attributes ~= 
sAttributeReflection(to!string(a), typeof(a).stringof);`);}



Note that all this code is simply trying to get the attributes. 
Notice how easy it is for a the base type:


// Get the attributes for the type
static if (__traits(compiles, __traits(getAttributes, T)))
	static foreach(a;  __traits(getAttributes, T)) Attributes ~= 
sAttributeReflection(to!string(a), typeof(a).stringof);


But because there is no `T` for fields we can't do it so easy, so 
we have to construct the name and use string mixins. But things 
start to break down here. One would think that we could just do

// Get the attributes for the field
import mModel;
mixin(`static foreach(a;  __traits(getAttributes, 
`~T.stringof~`.`~name~`)) Attributes ~= 
sAttributeReflection(to!string(a), typeof(a).stringof);`);

which is just the same code except we have to import the model 
because now we are using it directly(not as a type but as 
someclass.somefield).

This code then breaks down for private fields!

mReflect.d-mixin-201(201): Deprecation: 
`mModel.cDerived!int.cDerived.testField2` is not visible from 
module `mReflect`

Why? because we are trying to access it directly and it is 
private, although it is at compile time, we are in CTFE so the 
protection scheme prevents us from accessing.


for example, this code works:
mixin(`import `~moduleName!T~`;`);	
static if (name != "")
	mixin(`static foreach(a;  __traits(getAttributes, 
`~T.stringof~`.`~name~`)) Attributes ~= 
sAttributeReflection(to!string(a), typeof(a).stringof);`);

Notice how much more "complicated"(rather, messy and error prone) 
it is... just because we have to construct the field.

Also that it fails for private fields, it works here because it's 
only depreciated, but later on for methods we get an error 
instead!


Fields are relatively simple because they are just a name, 
attributes, and a type.

Here are the issues though:

Because their is no Field Type:

1. We have to hack things together using string mixins.

2. It requires importing the module that contains the type just 
to reflect on it. This then creates protection issues and 
possible name collisions(although we can alias the module)

3. We have to pass around the parent type and field in to 
templates and construct the fields by hand. This makes then makes 
a non-unified system. Notice how much easier it is to deal with 
when we just have a field type. It's not just the 2 line of code 
but also how it integrates with the rest of the code and the 
cBaseReflection. (and all the other templates such as mangledName 
and such)

Solution: Seems that adding a Field Type to D would completely 
solve all these issues. It is just another type who's main 
purpose is to encapsulate field as types making them "1st class".


----------

Now, moving on to methods, essentially the same problem. Because 
methods(and functions) are not types in and of themselves, it 
requires hacks like the above, but much worse since they are more 
complicated.


Some thing here, we gotta import the module that has the method, 
meaning same issues as above

mixin(`import `~moduleName!T~`;`);	// Must import the type to be 
able to inspect it(fragile and bizzare and ridiculous)
T t = T.init; // left over from trying to get things to work

Then, again, we can't call super.Reflect!T to do the work and 
most duplicate code:


Id = name;	
static if (__traits(compiles, TypeName = T.stringof)) TypeName = 
T.stringof~"."~name;
ModuleName = moduleName!(T);
FullName = fullyQualifiedName!(T)~"."~name;
MangledName = mangledName!T;
//mixin(`MangledName = mangledName!(`~T.stringof~`).`~name~`;`);


After messing with this stuff for a few days or so and it just 
getting becoming more obvious that it wasn't going to work well I 
decided to give up. While it may be possible to hack most things 
and get them to work.

For example,

mReflect.d-mixin-222(222): Error: no property `ValueProp` for 
type `string`


Of course, it is giving fields, so I have to modify the 
cAggregiateReflection code to only return appropriate methods:

		static foreach(k, m; __traits(derivedMembers, T))
		{{
			static if (__traits(compiles, __traits(getVirtualMethods, T, 
m)))
			static foreach(j, v; typeof(__traits(getVirtualMethods, T, 
m)))		
			{{
				//pragma(msg, T.stringof, " --- ", v.stringof);
				Methods ~= (new cMethodReflection()).Reflect!(T, m);

			}}
		}}

so it suggests we need a derivedMembers. I tried to use 
getVirtualMethods exclude fields but I guess it didn't work 
right, I see that I left the m in their instead of changing it to 
a v but v is the type of the method and not a name like m is.


So, the whole point of the Reflect library is to provide a 
uniform basis to get the desired information without all this 
nonsense(IMO).

It's not that it isn't necessarily doable but one can see that it 
requires a lot of unpleasant coding and really destroys the 
elegance of using oop and CTFE to solve the problem. If I have to 
duplicate the code in each class and use string mixins then what 
really is the point? (one could use a single template and string 
mixins to solve that but it becomes such a mess that it isn't fun 
and makes me wonder what other kinda nonsense will I run in to in 
the future?)


My suggestions:

Every D concept should have D type. attributes are types, unions, 
classes, fields, methods, enum members, symbols, expressions, 
code bodies, etc.  This is akin to Object in oop.

Every type template should then work find with all these types 
then. Want the module name of an attribute, just do moduleName!a 
where a is an attribute type returned by getAttributes.

This may not be feasible for all the things in D at least fields 
and methods and whatever other major ones that would be needed. 
(templates? I think they are covered under methods?)

Maybe the idea would be better served to be able to box up info 
to make a type such as DType and then be able to pass that 
around. In that case I could build the field and method types and 
then simply use Reflect and all the other easy code to get it to 
work.


Ideally the compiler itself would expose all the info available 
to a D program in a unified way. e.g., the Reflect library here 
would essentially be integrated(conceptually, not the code) in to 
the d compiler. This would replace the need for traits.

D could have a reflect keyword:

reflect!T

and one gets back a reflection type that contains all the info 
one wants in hierarchal form:

reflect!MyClass.Name
reflect!MyClass.Module.RootTypes
reflect!MyClass.Module.Imports
reflect!MyEnum.Uses      // Lets one get all the known uses of 
the enum
reflect!MyFunc.Body      // returns the code body(we can do this 
now by
importing the module as text and locating it but it is fragile)
reflect!MyFunc.Location  // returns the Source location(file, 
line, etc)
reflect!MyAttribute.New(Name = MyOtherAttribute, Value = 434) // 
Lets one construct a new type based off the old one, maybe not a 
great idea)
reflect!MyClass.New(Name = Name~Other, Body = { })
etc...

The goal of the code I wrote was to provide as much info as I can 
get in a unified way(as I've said many times already). Since it 
just wraps traits there is no drawback that I can think of and 
there shouldn't be(although since it uses classes it might not be 
usable in all areas of D).

If those in the know got serious about this idea it could provide 
a very powerful and elegant solution for D's meta system that 
will heavily complement it(it would also be easy to extend since 
one just ads to the hierarchy and all the internals are 
abstracted away).

I use meta programming heavily in D and that is the main thing I 
like about it.  Being able to write generic code means being able 
to work in general terms. This then requires the ability to 
introspect in the type system. D has the abilities but the 
conceptualization is very messy.  For one, there is __traits and 
then there is std.traits.  One could argue that one can wrap 
everything in __traits but that is easier said than done(as I 
have proven).

One problem with a "library" solution is speed. The compiler 
already parses all this information once and by having a internal 
reflect! the data can be filled out once during compilation and 
the using it is just looking up that data(at the cost of memory, 
could be lazy though).

I'm not sure the cost of repeatedly calling traits functions, if 
they are essentially memorized or if it requires some 
calculation, but if it does require any significant time, I 
imagine that a reflect! would then drastically increase 
compilation times when heavy meta programming is involved) as 
most introspection's would just be a look up.

Anyways, hopefully you guys can come up with something usable. I 
find that D's meta programming is very capable but also can be 
very tiresome to do simple things. I find myself having to use 
string mixins quite often which makes it very difficult to debug 
and read but also seems extremely unnecessary. The same patterns 
are repeated over and over and over.

While much of this work may be long term(D3?)[although, to be 
honest, it didn't take me that long to through the code I wrote 
together. Most is just wrapping. It's the anomalies of traits 
that caused the problem] I'd hope that at the very least a good 
solution to the main problems of fields and methods/functions in 
this library could be created.

Thanks.