Big picture on shared libraries when they go wrong, how?

[Richard (Rikki) Andrew Cattermole]

On 15/05/2024 6:19 AM, Atila Neves wrote:
> On Friday, 10 May 2024 at 15:21:15 UTC, Richard (Rikki) Andrew 
> Cattermole wrote:
>> On 11/05/2024 1:27 AM, Atila Neves wrote:
>>>>> "you cannot tell the compiler that a module is not in your binary." 
>>>>> - isn't this exactly what happens with `export` on a declaration 
>>>>> (as opposed to a definition)? That is, my understanding is that 
>>>>> `export` with no body means `dllimport` and `export` with a body 
>>>>> means `dllexport`.
>>>>
>>>> Keep in mind that nobody that I know of is using it to mean this today.
>>>
>>> Maybe the recommendation should then be that they should? Doesn't the 
>>> point still stand that "you cannot tell the compiler that a module is 
>>> not in your binary" isn't actually true? I saw in one issue where 
>>> there was a problem with variable declaration though, where 
>>> dllimport/dllexport was determined by the presence or not of an 
>>> initialiser, which is... yuck.
>>
>> "you cannot tell the compiler that a module is not in your binary" is 
>> true, there is no syntax or cli flag to do this today.
> 
> Correct, sorry, I thought we were talking about symbols and somehow 
> missed "module". The question would then be: why would someone want to 
> tell the compiler that an entire module is out of binary?

1. Metadata (ModuleInfo, TypeInfo, RTInfo)
2. It is almost certainly going to be correct and ``-I`` is almost 
certainly at play, therefore we can leverage this information to turn 
``export`` for positive annotation into ``DllImport``.

Can you spot the problem with this: ``[void*]`` vs ``[void*, void**, 
void*]``

Metadata quite literally is incapable of crossing the DLL boundary 
without knowing if its in or out of binary. It prevents linking.

If it does link, that pointer could very well be pointing at a jump 
instruction. Not exactly a fun day if you want to debug it.

>> But yes having an initializer or not, should not determine the symbol 
>> mode. I did my best to clean it up in a way that would keep things 
>> simple and not break the world.
>>
>> This is why I've simplified things down to:
>>
>> Use ``export`` + ``extern`` to go into ``DllImport`` mode.
> 
> Makes sense.

This also plays into the external import path switch, since we know that 
is almost certainly correct, adding the extern on make a very clean 
simple solution for positive annotation!

>> Or rely on the external import path switch to set the ``extern`` for 
>> the majority of users. Which is ideal for things like the di generator 
>> or build managers ;)
> 
> Do you mean "build systems"?

They are interchangeable at this level in my mind, but yes.

>> Or use the dllimport override switch to set all symbols found from a 
>> module that is known to be out of binary as ``DllImport`` (helps with 
>> mixing some imports being in binary and some out).
>>
>> Yeah that's not the best example, but keep in mind that symbol is not 
>> in ``DllImport`` mode, its ``Internal``.
>>
>> And that right there is a problem.
> 
> And wouldn't the solution be to add `export`?

Both druntime and PhobosV2 will remain in using negative annotation for 
the foreseeable future. The amount of work to convert that to positive 
is quite significant because its not just slapping export on things, you 
also have to test it.

Now do the rest of the ecosystem but with people who don't know what 
they are doing.

>>> Why would the entire module be dllimport?
>>
>> Walter came up with this idea a while back, so I'm a tad defensive 
>> towards it.
>>
>> https://github.com/dlang/dmd/pull/15298
> 
> I went through all of that and am still confused as to whether you want 
> or don't want whole modules to be declared out of binary, or why one 
> would want to do that.

Being out of binary is what that PR does, by deriving it based upon any 
symbol being in DllImport mode.

We need to know if it is out of binary (see at top of this comment 
especially here), and will give false positives if you use positive 
annotation due the deriviation.

>> With Walter's idea the ``app`` module constructor could be called 
>> before ``binding``'s does, which absolutely should not happen.
> 
> I don't know why not, nor did I understand the relevance of the example.

It shows that shared libraries weren't involved, yet because a module 
was derived as out of binary things didn't work correctly.

>>> My question is: when would I want to export a private symbol?
>>
>> Q1: Should a given symbol be private, yes?
>> Q2: Does any template use the previous symbol?
>>
>> A: That is why you would export a private symbol.
> 
> Can't we do what C++ does and stick the private symbol where the 
> template is being instantiated?

So you want even more global state?

And why am I feeling like there is a shifting sands feeling going on for.

>> ```d
>> module database_access;
>>
>> @safe:
>>
>> void doAThing(T)() {
>>     iCanKillYourDatabase(false, T.sizeof);
>> }
>>
>>
>> /*private:*/
>> export:
>>
>> void iCanKillYourDatabase(bool doWrongThing = true, size_t sizeOfThing) {
>>     if (doWrongThing)
>>         database.corruptSilently();
>> }
>> ```
>>
>> A bit dramatic (due to unrealistic nature), but that should get the 
>> point across that ``iCanKillYourDatabase`` should be private but also 
>> exported.
> 
> Inline it instead? The code is right there.

What if its accessing global state (and perhaps giving you access to it 
via callback)?

What if it is global state instead of a function?

What makes you think that it can be inlined?

>> For generated symbols like ``T.init``, ``opCmp``, ``ModuleInfo`` ext., 
>> we have zero control over these currently. Either you use a linker 
>> script or you use negative annotation. Everyone I know of uses 
>> negative annotation (although I support both).
> 
> What would the solution be?

Some cannot be like ``ModuleInfo``, others have be exported based upon 
if other things in the encapsulation are exported.

Its either that or we export literally everything inside of the 
encapsulation unit (I don't like that at all).

Or we invent new syntax... Again not a fan.

>>>>> "By not exporting ModuleInfo and assuming it is available the 
>>>>> compiler introduces a hidden dependency on a generated symbol that 
>>>>> may not exist." - do we have an issue for that? I searched for 
>>>>> ModuleInfo in the issues but none of them looked like a match to me.
>>>>
>>>> Yes two. They are referenced in the article.
>>>>
>>>> Note: they are not duplicates.
>>>>
>>>> Okay I lie there is a bunch more.
>>>
>>> Thanks!
>>>
>>> On a somewhat related note, we use dlls at work and seem to have 
>>> fixed "everything" by using ldc and `-fvisibility=hidden 
>>> -dllimport=defaultLibsOnly`, as well as `-link-defaultlib-shared`.
>>
>> ``-link-defaultlib-shared`` sets the druntime to be a shared library 
>> (which lets face it should be the default).
> 
> Given how much people go on about how great Go is because it links 
> statically (even though C/C++ have been able to do this for basically 
> forever if you opt-in), I'm not sure of that.

The way I view it is as thus:

- If you use D shared libraries with a static runtime, you're going to 
have your program have indeterminate behavior.
- On the other hand, if you don't use shared libraries with a shared 
runtime it works.

In the latter you will need to copy the druntime/phobos shared library, 
but hey the system loader will tell you if you forgot!

In the former there is no warning, it will happily do the wrong thing 
with no warnings and it might not even crash it could just corrupt data 
instead.

So from my perspective its better to be opt-in for static runtime/phobos 
builds if you know you don't need it, than the opposite.