COM Expertise needed: COM Callbacks

[Nierjerson via Digitalmars-d-learn]

On Tuesday, 25 April 2017 at 10:03:30 UTC, Atila Neves wrote:
> On Monday, 24 April 2017 at 00:55:45 UTC, Nierjerson wrote:
>> Still trying to get the com automation code working. This is a 
>> general issue with COM programming as I do not have the 
>> experience to solve the problem.
>>
>> [...]
>
> I tried looking at this because I just did some COM work even 
> if most of it in C++. There's a _lot_ of code and I was 
> instantly lost, even with your explanations.

There isn't a lot of code! All the code in Gen is irrelevant 
except for a few functions... even then, you don't ever really 
need to mess with it except to see a few of the definitions.

99.9% of the code is irrelevant, I could have removed most of it 
and it would function the same. The reason why I left it in is 
for completeness and because many of the different classes 
reference other classes so having one sorta requires having all 
the rest(even though they are never used, it is to avoid missing 
symbols, etc, which can be fixed but would require some editing 
and would prevent the code from working).

Only three classes are used is used in Gen.d: cApplication, 
cSolidColor, and cRGBColor.

One only really needs to look at main.d. I've put all the code 
necessary in that file. One only needs to look at Gen.d to get 
the general idea and at that, only those specific classes.




> I'd forget about Photoshop for now, just write a simple COM 
> client. That'd also ease in getting other people to help 
> because having Photoshop is a high barrier for entry.

Photoshop is the main app I'm trying to get to work so this is 
appropriate for me and it is also the only real way I have to 
test COM.



> Can you get this to work in C++?

The code generation probably could be tailored to work in C++ 
without too much issue. I'm not going to do it though! Too much 
work as it is and I first need to get this to work. I don't see 
any real difference between D and C++(if I'm going to use C++ for 
anything I'm going to use D instead). I've already had to rewrite 
the engine because D's CTFE could not handle the load. So this 
has turned out to be quite a bit of work.


---

The main issue has nothing to do with PS though(I imagine it is a 
general COM programming task).

The server(PS in this case) gives me an interface that has 
functions like RGB(cRGBColor rgb); (Actually a _RGBColor com 
interface which I wrap with a cRGBColor interface to provide the 
ability to handle the marshaling and such behind the scenes),


So I pass it a cRGBcolor interface, which actually can't be used 
because photoshop has no clue about that type.

Instead, I construct a dummy interface that inherits from 
IDispatch which I'll pass to server instead. I'll use it as a 
wrapper to hook up everything I need.

The problem is, when I do this, the app crashes. I do not know 
why. What I do know is that it seems to be calling QueryInterface 
and so it seems like it might actually be getting to the server 
and the server is calling the QueryInterface method. If that is 
the case, then the passing of the interface is working but 
something else is failing.

I'm just not sure though as I don't know enough about COM to 
determine what exactly is going on(after all, it's all happening 
on the server "behind closed doors").



....

	auto rgb2 = new icRGBColor();
	rgb2.iDispatch = rgb2;
	//auto rgb = cast(icRGBColor)&rgb2;
	//dd.RGB(cast(cRGBColor)(cast(void*)rgb));
	//dd.RGB(cast(icRGBColor)(cast(void*)rgb));


	cSolidColor.RGBSet = &RGB;
	auto rgb = new icRGBColor();
	//auto rgb = cast(icRGBColor)(cast(void*)dd.RGB());
	dd.RGB(rgb); // <--- This is where the work begins! Tries to get 
photoshop to set the RGB color through the rgb object


     return 0;
}

// This is the invoker.Only useful for debugging(brought out from 
Gen.d here for convenience
// The invoke line is what is important. When it is called, 
photoshop will be given the icRGBColor object and will do things 
with it(if it works, that is... seems to and seems to call 
QueryInterface but doesn't get any further).
void RGB(icRGBColor ic, cSolidColor s)
{
	import main;
	EXCEPINFO exception;
	uint argErr = 0;
	auto iidNULL = IID_NULL;
	auto RT = new SafeVariantPtr();
	VARIANT[1] paramVars;
	DISPPARAMS params = {rgvarg: paramVars.ptr, cArgs: 1, 
cNamedArgs: 0};
	auto ID = s.COMMethodIDs[`RGB`];
	paramVars[0].punkVal = ic; paramVars[0].vt = 
VARENUM.VT_DISPATCH; scope(exit) VariantClear(&paramVars[0]);
	auto res = s.iDispatch.Invoke(cast(int)ID, &iidNULL, 0, 
DISPATCH_PROPERTYPUT, &params, cast(VARIANT*)RT, &exception, 
&argErr);
	assert(res == S_OK, `Could not invoke COM Function 
cSolidColor.RGB. Error `~to!string(res, 16));

}



...

The icRGBColor interface that follows is a simple callback 
interface. All the methods simply print their function name when 
called, this helps to know when they are called.

When one runs the program they should see a few calls to 
QueryInterface. Either photoshop is calling this method on 
purpose or by accident.(calling something else but the vtable is 
wrong)


Hope that helps some. I only need some advice, guidance on how 
things are suppose to work so I don't wait a lot of time going 
down a blind alley.