should a delegate context ever be a non pointer?

[BCS]

A while ago I was writing some code and ended up with a template 
something like this:

struct S
{
	int i;
	// other stuff
}

S FillS(int j, int k)(MyObject o)
{
	S ret;
	o.something(k);
	ret.i = j;
	// fill other stuff

	return ret;
}

I was building large tables of function pointers from this template. All 
was well and good when I only had one version of it actual used. Then I 
actually used it with the 40+ versions that actually exist. It made my 
code run about 100% longer (I'm calling these things a lot).

So then I looked around for a better way to do it and realized that 
there  was less than 32 bits of difference between the versions (i and k 
were small) so I switch to this:

union U
{
	S2* pt
	struct
	{
		ushort h;
		ushort l;
	}
}

struct S2
{
	S FillS(MyObject o)
	{
		U u;
		S ret;

		u.pt = this;
		o.something(u.h);
		ret.i = u.l;
		// fill other stuff

		return ret;
	}

	static delegate(MyObject) Build(uint i, uint k)
	{
		U u;
		u.h = k;
		u.l = i;
		return &u.pt.FillS;
	}
}

Most of the performance came back. However, That is just some funky 
programming.

My question is, is there any reason that such a things shouldn't be 
endorsed? What that would requirer is that the context in a delegate be 
defined to be an opaque field and allowed to be anything that fits in a 
size_t. Then the above would become:



struct Ctx
{
	ushort h;
	ushort l;
}


S FillS(Ctx c, MyObject o)
{
	S ret;
	o.something(c.h);
	ret.i = c.l;

	// fill other stuff
	return ret;
}

static delegate(MyObject) Build(uint i, uint k)
{
	Ctx c;
	c.h = k;
	c.l = i;
	return &c.FillS;
}