Ideas for runtime loading of shared libraries.

[Johannes Pfau]

Am Thu, 07 Mar 2013 12:13:03 +0100
schrieb Martin Nowak <code at dawg.eu>:

> On 01/02/2012 08:20 PM, Martin Nowak wrote:
> 
> I now think we should do the initialization directly when loading,
> even though that might cause deadlocks in certain scenarios. It would
> solve all mentioned initialization issues. This will also serialize 
> initialization on the loader mutex but I don't expect this to become
> a performance issue.

That's the approach used by other runtime libraries, right? So it
should work in most cases.

> 
> >   - Getting symbols through mangled names is difficult/unstable.
> >

With D methods the problem is that the mangled name contains the module
name, so it'll change for every plugin. This is something a reflection
library has to solve:

@plugin("testFunction") void testFunction(){};

auto lib = SharedLib.load();
reflect.getFunction!(void function())("testFunction", lib);

Where the reflection lib would have to build a name->function pointer
map in the shared library and export a C function "void*
getFunction(string name)" which can be accessed via dlsym. (Some more
work could be done to ensure type safety)
(or we just use extern(C) functions)

For OOP features it might be nice if we can iterate all classes that
implement an Interface

foreach(IWebPage page; lib)

Although the question then is how to construct instances of these
classes. Maybe the factory pattern is better for this.

> 
> >   - Libraries might not be unloaded as long as GC collected class
> > instances still exist because
> >     finalization fails otherwise.
> >
> 
> We should extend the GC to allow marking and finalizing of foreign 
> memory. This is needed for shared memory too.

Would it be possible to implement this in a way that shared libraries
can be used without the GC in code which doesn't use the GC (of course
with a loss of convenience & safety?)

For example we could introduce this API:

/**
 * Force unload library. All types contained in this library must have
 * been finalized by now  
 */
@system void forceUnload(SharedLib l);  

/**
 * Internally called by the gc on every scan
 */
@safe private void gcCheckUnload()
{
   foreach(SharedLibrary l; loadedLibs)
     if(shouldCollect(l))
         forceUnload(l);
}

/**
 * Optional additional method to force unloading with GC help:
 * Finalize all types from this SharedLib, then unload. If user
 * code still keeps a reference to a type from this lib it'll crash.
 */
@system void forceGCUnload(SharedLib l)
{
    foreach(void* instance; objects)
    {
        if(inLib(instance), l))
            rt_finalize2(instance);
    }
    forceUnload(l);
}  

As far as I understand the issue, C is necessary if we use the GC but
want to force-unload a library: Even if user code no longer references
types from the shared lib the GC might not have ran the finalizers yet.
So this methods finalizes all objects from the shared lib (even if the
user still keeps a reference) and then unloads the lib.

I think this is useful in a environment with a massive number of
loaded / unloaded plugins. For example webpages could be written as
plugins and the webserver could load/unload those. The webserver has to
make sure that the libraries are really unloaded to free some
resources (AFAIR mono/asp.net implements webpages as such plugins) 

> 
> > Any ideas or use-cases for plugins are welcome.
> >
> 
> Still holds.

Use cases?

* Web pages as plugins
* video/audio filtering framework (avisynth, gstreamer)
* syntax highlighting for unknown languages in text editor
* adding support for a new languages / tools to IDEs 
* Filters in image editing tools
* plugins for webbrowsers (adblocker, dom manipulation)
* Adding new import / export formats to almost any kind of
  editor / viewer
* Adding web site / media information scrapers to media players (xbmc,
  rhythmbox)