Big picture on shared libraries when they go wrong, how?

[Atila Neves]

On Thursday, 16 May 2024 at 02:15:33 UTC, Richard (Rikki) Andrew 
Cattermole wrote:
>
> On 16/05/2024 5:30 AM, Atila Neves wrote:
>> On Tuesday, 14 May 2024 at 22:13:19 UTC, Richard (Rikki) 
>> Andrew Cattermole wrote:
>>>
>>> On 15/05/2024 6:19 AM, Atila Neves wrote:
>>>> [...]
>>>
>>> 1. Metadata (ModuleInfo, TypeInfo, RTInfo)
>> 
> All three of those are examples of things that because they 
> exist do not link, load or run at runtime correctly.

I still don't understand what are the situations under which 
things don't work. Or why they require marking an entire module 
as out of binary.

> These files should not exist:

Oof. No, they should not. I understand you have workarounds here 
for problems with shared libraries in D, but I still don't 
understand why these workarounds are necessary.

>>> 2. It is almost certainly going to be correct
>> 
>> What is most certainly correct?
>
> If a module is found via a path provided to the external import 
> path switch to be out of binary.

This "begs the assumption" that such a switch exists.

>>> and ``-I`` is almost certainly at play, therefore we can 
>>> leverage this information to turn ``export`` for positive 
>>> annotation into ``DllImport``.
>> 
>> We already have a mechanism to do this.
>
> Ugh no? Not for positive annotation we do not.

We do?

     export void foo();    // dllimport
     export void foo() { } // dllexport

>>> Can you spot the problem with this: ``[void*]`` vs ``[void*, 
>>> void**, void*]``
>> 
>> No, but I don't understand the question.
>
> ModuleInfo goes into an array during loading.
>
> For ones in binary that is an array of them.
> [...]

I get an inkling of what the problem is here with the 
description, but:

* I don't actually know
* I have a gut feeling there's a solution that isn't too 
complicated

> Its not like how it is when its all in binary. This is not 
> something any user of D should be needing to attempt a work 
> around for and **I have working ones** puke.

I agree.

>> I don't see how `extern` is related to a potential external 
>> import path switch.
>
> Simple, it adds it.

Right, but why does it need to?

     extern export int foo; // dllimport
     export int foo;        // dllexport

> This is desirable when you are doing multiple step builds, such 
> as incremental compilation or ya know using static libraries 
> like dub requires today.

We do this at work everyday; I don't get why the switch would be 
needed.

>> What needs to be tested?
>
> Anytime you convert a module from negative annotation to 
> positive, everything has to be tested to verify that it still 
> links, loads and runs correctly.

What's "everything"?

> In the above you were promoting duplicating global state as a 
> solution instead of letting the user mark something that is 
> private as exported.
>
> It's an insane idea.

It's what C++ does. There's no way to do it without putting it 
into a header and marking it `static`.

> Imagine there being multiple different mallocs inside each 
> shared library!

Because of a global variable?

>>> What makes you think that it can be inlined?
>> 
>> C++.
>
> Well C++ isn't D.
>
> They have things we don't have, different expectations.

And we have things they don't have; but given that 
dll{import,export} is a Microsoft C/C++ thing, I think that it's 
a good idea to use them as inspiration for what we should do.

> Keep in mind that D's official build manager is based upon npm, 
> not CMake. We use D very differently than C++ users use C++.

I don't see how this is relevant, especially since there are 
multiple (meta) build systems one can use for both.

> You have to understand how D is being used here

Ok, how is it being used in a way that's so different from how 
C++ is used?

> That is negative annotation. It is the default in dub

How is it the default in dub?

> Defaults should never be foot-gun heavy.

I agree.

> That is what you are promoting here by suggesting the status 
> quo with static library builds of druntime/phobos are a good 
> thing.

I don't see how; 0 bullets in my foot due to this so far in 10 
years.

> It's why C always defaults to shared libraries for its libc and 
> with that the C++ runtime as well.

I'm 99.9% sure that it was to save space and for every 
dependendent app to be updated automatically to the new version 
of the libraries.

>>> So from my perspective its better to be opt-in for static 
>>> runtime/phobos builds if you know you don't need it, than the 
>>> opposite.
>> 
>> Again, the raves from people new to Go would suggest 
>> otherwise. So would the lack of requests for this in D.
>
> Go isn't D, its used in a very different way to D is.

Yes/no. I'd rather statically link, personally.

> You are very much mistaken about lack of requests, they are not 
> needed because dub automates the change of defaults.

I'm confused; are there requests or not?

> When I first introduced support for shared libraries into dub, 
> you had to explicitly set that you wanted a shared library 
> build of druntime/phobos as well as setting dllimport to all 
> and visibility to public.

Ok. And now it automatically says "dynamically link"? If that's 
the case, I didn't know that, and now I'm wondering why we have 
an explicit flag for that in a dub.sdl.