Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)

[Paulo Pinto]

On Monday, 20 August 2012 at 07:26:40 UTC, Jacob Carlborg wrote:
> On 2012-08-20 05:40, Carl Sturtivant wrote:
>> (By dynamic loading I mean using something like the C library 
>> function
>> dlopen to explicitly load a shared object at runtime. I do not 
>> mean
>> dynamic linking in the usual passive sense.)
>>
>> I just successfully got working a toy example of dynamically 
>> loading
>> (from a D program) a D shared object and then finding and 
>> calling
>> functions defined in the shared object, and not just with C 
>> linkage.
>>
>> The main program, main.d, was compiled and linked somewhat 
>> normally,
>> except for passing the linker via gcc the flags necessary to 
>> ensure that
>> the whole of libphobos2.a is present and that all symbols in 
>> the
>> resulting executable are exposed dynamically.
>>
>> The shared object source, dload.d was compiled to an object 
>> file
>> containing position independent code by dmd. Then I invoked 
>> the linker
>> explicitly and had it make a shared object without the D 
>> runtime system
>> or Phobos. This is the novel step, and it enables the shared 
>> object to
>> resolve its linkage to the D runtime system and Phobos at the 
>> point of
>> being loaded, via callbacks to the main program. Thus the 
>> troubles of D
>> in shared objects are largely circumvented. There is only one 
>> instance
>> of phobos and D-runtime, in the main program. (Once phobos and 
>> druntime
>> are shared objects in the future somewhere this will work with 
>> no code
>> bloat.)
>>
>> The static initialization code in dload.d is automatically 
>> executed when
>> the shared object libdload.so is loaded by the main program, 
>> because the
>> linker is also passed a flag indicating the static 
>> initialization
>> block's mangled name, dynamically determined from dload.o 
>> before linkage
>> to libdload.so occurs.
>>
>> Finally, the mangled names of the functions to load are 
>> determined by a
>> call of a function with C linkage in dload.d from main.d that 
>> looks up
>> those names in an associative array initialized in the static
>> initialization block of dload.d where those mangled names are 
>> directly
>> available, so that full D linkage can be emulated, at least 
>> for functions.
>>
>> One thing: the garbage collector needs to be aware of static 
>> and
>> 'global' D variables in the shared object. Can a technical 
>> expert verify
>> that I've done the right thing to achieve that happy state of 
>> affairs in
>> this unusual context?
>>
>> So, what's overlooked here? I know that the static 
>> initialization code
>> cannot successfully throw an exception. Yet if a function in 
>> the shared
>> object is called from the main program and throws an 
>> exception, all is
>> well. (Try these.) See my comments in dload.d about this. What 
>> is it
>> about the implementation of exceptions that's problematic here?
>>
>> All files attached, including a Makefile with the exact 
>> options passed
>> to dmd, gcc and ld.
>
> I'm not sure I'm following what you exactly have done here but 
> in general this is what needs to be done to make dynamic 
> libraries properly work in D :
>
> * Initialize module infos (module constructors and similar)
> * Add TLS variables
> * Add exception handling tables
> * Add GC roots
>
> The above four things need to be extracted from the loaded 
> dynamic library and it gets loaded and preferably remove them 
> as well when the dynamic library gets unloaded. Currently this 
> is only extracted from the running executable. This is platform 
> dependent but usually it's extracted using bracketed sections 
> via extern C variables.

Should this be made automatically by the compiler?

This would be my expectation based on my experience with dynamic 
libraries in Turbo Pascal/Delphi.

--
Paulo