Dynamically binding to D code using extern(D)

jfondren julian.fondren at gmail.com
Thu Sep 30 22:30:30 UTC 2021


On Thursday, 30 September 2021 at 18:09:46 UTC, Hipreme wrote:
> I write this post as both a learning tool, a question and an 
> inquiry.
>
> There are just a lot of drawbacks in trying to do function 
> exporting while using D.

The terms that people use are a bit sloppy. There are three kinds 
of 'linking' here:

1. static linking, performed during compilation, once. If linking 
fails, the compile files.
2. dynamic linking (option 1), performed when an executable 
starts up, before your program gains control, by the system 
linker. If linking fails, your program never gets control.
3. dynamic linking (option 2), performed arbitrarily at runtime, 
by your program. If linking fails, you can do whatever you want 
about that.

All of the loadSymbol and 'userdata module' hassle that you're 
frustrated by is from option 2. Option 1 is really the normal way 
to link large shared libraries and there's nothing to it. What 
your code looks like that loads a shared library is just `import 
biglib;`, and the rest of the work is in dub, pkg-config, 
`LD_LIBRARY_PATH`, etc. Phobos is commonly linked in this way.

Pretty much anything that isn't a plugin in a plugin directory 
can use option 1 instead of option 2.

> extern(C) advantages:
>
> - Code callable from any language as it is absolutely intuitive
> - Well documented
>

You can call scalding water 'hot' even when you're fresh from 
observing a lava flow. People still find the C ABI frustrating in 
a lot of ways, and especially when they encounter it for the 
first time.

But the C ABI rules the world right now, yes. The real advantages 
are

- it 'never' changes
- 'everyone' already makes it easy to use

> extern(C) disadvantages:
>
> - You will need to declare your function pointer as extern(C) 
> or it will swap the arguments order.

- you're limited to using C's types
- you can't use overloading, lazy parameters, default values; you 
can't rely on scope parameters, etc., etc.
- you can't casually hand over GC-allocated data and expect the 
other side to handle it right, or structs with lifetime functions 
that you expect to be called
- very little of importance is statically checked: to use a C ABI 
right you need to very carefully read documentation that needs to 
exist to even know who is expected to clean up a pointer and how, 
how large buffers should be. (I wasn't feeling a lot of the C 
ABI's "absolute intuitiveness" when I was passing libpcre an 
ovector sized to the number of pairs I wanted back rather than 
the correct number of `pairs*3/2`)

Option 2 dynamic linking of D libraries sounds pretty 
frustrating. Even with a plugin architecture, maybe I'd prefer 
just recompiling the application each time the plugins change to 
retain option 1 dynamic linking. Using a C ABI instead is a good 
idea if just to play nice with other languages.

And if you were wanting something like untrusted plugins, a way 
to respond to a segfault in a plugin, like I think you mentioned 
in Discord, then I'd still suggest not linking at all but having 
separate applications and some form of interprocess communication 
(pipes, unix sockets, TCP sockets) instead of function calls. 
This is something that you could design, or with D's reflection, 
generate code for against the function calls you already have. 
But this is even more work that you'll have to do. If we add "a 
separate process telling you what to do with some kind of 
protocol" as a fourth kind of linking, then the respective effort 
is

1. free! it compiles, it's probably good!
2. free! if the program starts, it's probably good!
3. wow, why don't you just write your own loadSymbol DSL?
4. wow, why don't you just reimplement Erlang/OTP and call it 
std.distributed? maybe protobufs will be enough.


More information about the Digitalmars-d-learn mailing list