Big picture on shared libraries when they go wrong, how?

Richard (Rikki) Andrew Cattermole richard at cattermole.co.nz
Thu May 16 15:14:12 UTC 2024


On 17/05/2024 1:46 AM, Atila Neves wrote:
> On Thursday, 16 May 2024 at 02:15:33 UTC, Richard (Rikki) Andrew 
> Cattermole wrote:
>>
>> On 16/05/2024 5:30 AM, Atila Neves wrote:
>>> On Tuesday, 14 May 2024 at 22:13:19 UTC, Richard (Rikki) Andrew 
>>> Cattermole wrote:
>>>>
>>>> On 15/05/2024 6:19 AM, Atila Neves wrote:
>>>>> [...]
>>>>
>>>> 1. Metadata (ModuleInfo, TypeInfo, RTInfo)
>>>
>> All three of those are examples of things that because they exist do 
>> not link, load or run at runtime correctly.
> 
> I still don't understand what are the situations under which things 
> don't work. Or why they require marking an entire module as out of binary.

They are data not functions.

See Martin's post. 
https://forum.dlang.org/post/dobouzmhwabquswguunk@forum.dlang.org

They literally cannot cross the DLL boundary if you do not know that 
they are out of binary.

>> These files should not exist:
> 
> Oof. No, they should not. I understand you have workarounds here for 
> problems with shared libraries in D, but I still don't understand why 
> these workarounds are necessary.

Yeah I can tell, you're not quite getting the difference between data 
and functions here and unfortunately they matter if you are using 
positive annotation.

If I dropped positive annotation and went to negative I could get rid of 
them even for dmd (tested after Rainer added support).

>>>> and ``-I`` is almost certainly at play, therefore we can leverage 
>>>> this information to turn ``export`` for positive annotation into 
>>>> ``DllImport``.
>>>
>>> We already have a mechanism to do this.
>>
>> Ugh no? Not for positive annotation we do not.
> 
> We do?
> 
>      export void foo();    // dllimport
>      export void foo() { } // dllexport

This is no different than initializers on a global variable determining 
its symbol mode.

Think about the .di generator removing bodies, and suddenly you can no 
longer ship both static and shared libraries of your library.

One set of interface files to a library is better than two, one for each 
build.

>>>> Can you spot the problem with this: ``[void*]`` vs ``[void*, void**, 
>>>> void*]``
>>>
>>> No, but I don't understand the question.
>>
>> ModuleInfo goes into an array during loading.
>>
>> For ones in binary that is an array of them.
>> [...]
> 
> I get an inkling of what the problem is here with the description, but:
> 
> * I don't actually know
> * I have a gut feeling there's a solution that isn't too complicated

The only "easy" solution I know of is to nullify out if it can't load 
the symbol. That assumes the linker/loader supports it, and 
unfortunately as a solution it only solves ModuleInfo.

You still have to deal with other data like TypeInfo or .init.

Each time I had to deal with .init not crossing over it took me days to 
figure it out and here is my workaround to that:

https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/text/format/specifier.d#L262

https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/text/format/prettyprint.d#L28

>>> I don't see how `extern` is related to a potential external import 
>>> path switch.
>>
>> Simple, it adds it.
> 
> Right, but why does it need to?
> 
>      extern export int foo; // dllimport
>      export int foo;        // dllexport

What you have written is behavior my DIP promotes.

However the question is what happens when you have ``export int foo;`` 
in a file passed to ``-I``.

In my DIP that is in internal symbol mode, not ``DllImport``.

This allows for both static libraries and shared libraries to be used 
with one set of D files and in turn keeps the .di generator simpler.

>> This is desirable when you are doing multiple step builds, such as 
>> incremental compilation or ya know using static libraries like dub 
>> requires today.
> 
> We do this at work everyday; I don't get why the switch would be needed.

That's because you're building it as a plugin with a set exported interface.

This is a much more limited scenario that does not reflect projects like 
game engines or a standard library and should not be compared.

>>> What needs to be tested?
>>
>> Anytime you convert a module from negative annotation to positive, 
>> everything has to be tested to verify that it still links, loads and 
>> runs correctly.
> 
> What's "everything"?

All symbols that you change the symbol mode on.

>> In the above you were promoting duplicating global state as a solution 
>> instead of letting the user mark something that is private as exported.
>>
>> It's an insane idea.
> 
> It's what C++ does. There's no way to do it without putting it into a 
> header and marking it `static`.
> 
>> Imagine there being multiple different mallocs inside each shared 
>> library!
> 
> Because of a global variable?

Bad example, but yes.

If you are requiring duplicated globals that means you can end up with a 
library with N state, rather than 1.

With template instantiations that is a very real problem.

However I am now wondering if we might be miscommunicating on this.

>>>> What makes you think that it can be inlined?
>>>
>>> C++.
>>
>> Well C++ isn't D.
>>
>> They have things we don't have, different expectations.
> 
> And we have things they don't have; but given that dll{import,export} is 
> a Microsoft C/C++ thing, I think that it's a good idea to use them as 
> inspiration for what we should do.

They have no QoL stuff at this level for us to take inspiration from.

You use the macro preprocessor to set when something is DllImport versus 
internal.

As for terminology yes, we absolutely must migrate over. Everyone has. 
Both LLVM and GCC use it, it's the standard model at the compiler level now.

>> Keep in mind that D's official build manager is based upon npm, not 
>> CMake. We use D very differently than C++ users use C++.
> 
> I don't see how this is relevant, especially since there are multiple 
> (meta) build systems one can use for both.
> 
>> You have to understand how D is being used here
> 
> Ok, how is it being used in a way that's so different from how C++ is used?

Basically people expect to be able to import and for things to work.

Nobody should be seeing linker warnings when using dub just because they 
have a dependency and are building a shared library.

It is abnormal outside of plugins for people to be using positive 
annotation. They are exclusively negative annotation based upon the 
support I have given over a 2+ year period.

We have people in the community who do not wish to know that linkers 
exist and still manage (with help) to use shared libraries.

C++ on the other hand has for most of its life been exclusively positive 
annotation. Very different knowledge and willingness to deal with these 
details.

>> That is negative annotation. It is the default in dub
> 
> How is it the default in dub?

https://github.com/dlang/dub/blob/master/source/dub/compilers/ldc.d#L172

Its hard wired at the compiler personality level to change defaults.

Note: default on other platforms is negative annotation (which makes it 
highly inconsistent, which my DIP fixes).

>> It's why C always defaults to shared libraries for its libc and with 
>> that the C++ runtime as well.
> 
> I'm 99.9% sure that it was to save space and for every dependendent app 
> to be updated automatically to the new version of the libraries.

That is a nice side benefit. It also save RAM for the read only segments 
(that could be large due to tables).

But it has global state for things like malloc, that you really want to 
be unified within the entire application.

Otherwise you get memory leaks.

>> You are very much mistaken about lack of requests, they are not needed 
>> because dub automates the change of defaults.
> 
> I'm confused; are there requests or not?

I got it wrong, it wasn't dub that was changing defaults. Its ldc.

```
--link-defaultlib-shared                     - Link with shared versions 
of default libraries. Defaults to true when generating a shared library 
(-shared)
```

So no requests, because it is already switched (for shared libraries but 
not executables)!

>> When I first introduced support for shared libraries into dub, you had 
>> to explicitly set that you wanted a shared library build of 
>> druntime/phobos as well as setting dllimport to all and visibility to 
>> public.
> 
> Ok. And now it automatically says "dynamically link"? If that's the 
> case, I didn't know that, and now I'm wondering why we have an explicit 
> flag for that in a dub.sdl.

I got it wrong that its ldc doing it, although it did do it once back in 
2016.

I don't know why its there. Is that on an executable?




More information about the Digitalmars-d mailing list