Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)

[Jacob Carlborg]

On 2012-08-20 05:40, Carl Sturtivant wrote:
> (By dynamic loading I mean using something like the C library function
> dlopen to explicitly load a shared object at runtime. I do not mean
> dynamic linking in the usual passive sense.)
>
> I just successfully got working a toy example of dynamically loading
> (from a D program) a D shared object and then finding and calling
> functions defined in the shared object, and not just with C linkage.
>
> The main program, main.d, was compiled and linked somewhat normally,
> except for passing the linker via gcc the flags necessary to ensure that
> the whole of libphobos2.a is present and that all symbols in the
> resulting executable are exposed dynamically.
>
> The shared object source, dload.d was compiled to an object file
> containing position independent code by dmd. Then I invoked the linker
> explicitly and had it make a shared object without the D runtime system
> or Phobos. This is the novel step, and it enables the shared object to
> resolve its linkage to the D runtime system and Phobos at the point of
> being loaded, via callbacks to the main program. Thus the troubles of D
> in shared objects are largely circumvented. There is only one instance
> of phobos and D-runtime, in the main program. (Once phobos and druntime
> are shared objects in the future somewhere this will work with no code
> bloat.)
>
> The static initialization code in dload.d is automatically executed when
> the shared object libdload.so is loaded by the main program, because the
> linker is also passed a flag indicating the static initialization
> block's mangled name, dynamically determined from dload.o before linkage
> to libdload.so occurs.
>
> Finally, the mangled names of the functions to load are determined by a
> call of a function with C linkage in dload.d from main.d that looks up
> those names in an associative array initialized in the static
> initialization block of dload.d where those mangled names are directly
> available, so that full D linkage can be emulated, at least for functions.
>
> One thing: the garbage collector needs to be aware of static and
> 'global' D variables in the shared object. Can a technical expert verify
> that I've done the right thing to achieve that happy state of affairs in
> this unusual context?
>
> So, what's overlooked here? I know that the static initialization code
> cannot successfully throw an exception. Yet if a function in the shared
> object is called from the main program and throws an exception, all is
> well. (Try these.) See my comments in dload.d about this. What is it
> about the implementation of exceptions that's problematic here?
>
> All files attached, including a Makefile with the exact options passed
> to dmd, gcc and ld.

I'm not sure I'm following what you exactly have done here but in 
general this is what needs to be done to make dynamic libraries properly 
work in D :

* Initialize module infos (module constructors and similar)
* Add TLS variables
* Add exception handling tables
* Add GC roots

The above four things need to be extracted from the loaded dynamic 
library and it gets loaded and preferably remove them as well when the 
dynamic library gets unloaded. Currently this is only extracted from the 
running executable. This is platform dependent but usually it's 
extracted using bracketed sections via extern C variables.

-- 
/Jacob Carlborg