Big picture on shared libraries when they go wrong, how?

[Richard (Rikki) Andrew Cattermole]

On 16/05/2024 5:30 AM, Atila Neves wrote:
> On Tuesday, 14 May 2024 at 22:13:19 UTC, Richard (Rikki) Andrew 
> Cattermole wrote:
>>
>> On 15/05/2024 6:19 AM, Atila Neves wrote:
>>> [...]
>>
>> 1. Metadata (ModuleInfo, TypeInfo, RTInfo)
> 
> These are whats, not whys.

As always, this entire thread is all about why things do not link, load 
or run.

All three of those are examples of things that because they exist do not 
link, load or run at runtime correctly.

These files should not exist:

https://github.com/Project-Sidero/basic_memory/blob/main/msvc_exports.def

https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/dllmain.d

https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/moduleinfostubs.d

https://github.com/Project-Sidero/eventloop/blob/master/source/sidero/eventloop/rtinfoimplstub.d

https://github.com/Project-Sidero/image/blob/master/source/sidero/image/rtinfoimplstub.d

https://github.com/Project-Sidero/image/blob/master/source/sidero/image/dllmain.d

>> 2. It is almost certainly going to be correct
> 
> What is most certainly correct?

If a module is found via a path provided to the external import path 
switch to be out of binary.

>> and ``-I`` is almost certainly at play, therefore we can leverage this 
>> information to turn ``export`` for positive annotation into 
>> ``DllImport``.
> 
> We already have a mechanism to do this.

Ugh no? Not for positive annotation we do not.

>> Can you spot the problem with this: ``[void*]`` vs ``[void*, void**, 
>> void*]``
> 
> No, but I don't understand the question.

ModuleInfo goes into an array during loading.

For ones in binary that is an array of them.

Because we can only bind to the jmp wrapper function, or the DllImport 
global pointer /if/ we know that it is in another binary we end up with 
an array that has both ModuleInfo's and pointers to ModuleInfo's there 
is no way to reliably distinguish between the two.

>> Metadata quite literally is incapable of crossing the DLL boundary 
>> without knowing if its in or out of binary. It prevents linking.
> 
> What prevents linking?

Only functions can work across the DLL boundary, ROM data like arrays or 
meta data like ModuleInfo cannot cross directly.

If you know it is out of binary then you can perform a load against 
``ModuleInfo*``, but you cannot patch against ``ModuleInfo``.

Its not like how it is when its all in binary. This is not something any 
user of D should be needing to attempt a work around for and **I have 
working ones** puke.

>>> [...]
>>
>> This also plays into the external import path switch, since we know 
>> that is almost certainly correct, adding the extern on make a very 
>> clean simple solution for positive annotation!
> 
> I don't see how `extern` is related to a potential external import path 
> switch.

Simple, it adds it.

By adding it you convert an ``export`` to ``export`` with ``extern`` 
which the DIP defines as ``DllImport``.

An ``export`` by itself would remain in ``internal`` mode.

This is desirable when you are doing multiple step builds, such as 
incremental compilation or ya know using static libraries like dub 
requires today.

>>> [...]
>>
>> Both druntime and PhobosV2 will remain in using negative annotation 
>> for the foreseeable future. The amount of work to convert that to 
>> positive is quite significant because its not just slapping export on 
>> things, you also have to test it.
> 
> `export:`?
> 
> What needs to be tested?

Anytime you convert a module from negative annotation to positive, 
everything has to be tested to verify that it still links, loads and 
runs correctly.

>> Now do the rest of the ecosystem but with people who don't know what 
>> they are doing.
> 
> I think it's unlikely that most projects would need to work in dlls.

It doesn't matter, it only takes one project that depends upon another 
for the dependency to have to care about it.

>>> [...]
>>
>> So you want even more global state?
> 
> ?

In the above you were promoting duplicating global state as a solution 
instead of letting the user mark something that is private as exported.

It's an insane idea.

Imagine there being multiple different mallocs inside each shared library!

Lots more ram usage, and plenty of free's not freeing because the state 
wasn't shared.

And that assume you even succeed at making that link at all.

>>> [...]
>>
>> What if its accessing global state (and perhaps giving you access to 
>> it via callback)?
>>
>> What if it is global state instead of a function?
>>
>> What makes you think that it can be inlined?
> 
> C++.

Well C++ isn't D.

They have things we don't have, different expectations.

Keep in mind that D's official build manager is based upon npm, not 
CMake. We use D very differently than C++ users use C++.

You have to understand how D is being used here, there is not substitute 
for experience supporting people with shared library support and seeing 
what they are trying to do.

>>> [...]
>>
>> Some cannot be like ``ModuleInfo``, others have be exported based upon 
>> if other things in the encapsulation are exported.
> 
> What's an "encapsulation"?

struct/class/union

Basically anything that contains other symbols.

>> Its either that or we export literally everything inside of the 
>> encapsulation unit (I don't like that at all).
> 
> ?

That is negative annotation. It is the default in dub and is the only 
reliable method of using shared libraries in D and even then its no 
where near as reliable as it should be as you cannot use (sub)packages.

>>> [...]
>>
>> The way I view it is as thus:
>>
>> - If you use D shared libraries with a static runtime, you're going to 
>> have your program have indeterminate behavior.
> 
> Which doesn't seem to be what happens most often.

Right, we don't default to that with dub when using shared libraries, 
because it can only ever introduce undesirable behavior.

>> - On the other hand, if you don't use shared libraries with a shared 
>> runtime it works.
> 
> It doesn't seem this is what most people want.

Defaults should never be foot-gun heavy. That is what you are promoting 
here by suggesting the status quo with static library builds of 
druntime/phobos are a good thing.

It's why C always defaults to shared libraries for its libc and with 
that the C++ runtime as well.

>> So from my perspective its better to be opt-in for static 
>> runtime/phobos builds if you know you don't need it, than the opposite.
> 
> Again, the raves from people new to Go would suggest otherwise. So would 
> the lack of requests for this in D.

Go isn't D, its used in a very different way to D is.

You are very much mistaken about lack of requests, they are not needed 
because dub automates the change of defaults.

When I first introduced support for shared libraries into dub, you had 
to explicitly set that you wanted a shared library build of 
druntime/phobos as well as setting dllimport to all and visibility to 
public.

The reason nobody has talked about it is because its done for them 
automatically when they use dub. If they use something else, then they 
probably know how to figure out what they need to do on their own (I've 
had no support requests for anything that isn't dub).