[Issue 15060] New: Can't load a D shared library first, then load a C shared library

[via Digitalmars-d-bugs]

https://issues.dlang.org/show_bug.cgi?id=15060

          Issue ID: 15060
           Summary: Can't load a D shared library first, then load a C
                    shared library
           Product: D
           Version: D2
          Hardware: All
                OS: Mac OS X
            Status: NEW
          Severity: major
          Priority: P1
         Component: druntime
          Assignee: nobody at puremagic.com
          Reporter: aliloko at gmail.com

Repost of a message in https://issues.dlang.org/show_bug.cgi?id=14824
also reported here with a $50 bounty since I originally found it with LDC, but
it also happens with DMD: https://github.com/ldc-developers/ldc/issues/1071


DMD version = 2.068.1
OS = OS X 10.10.4

== Setup ==

Here the host program source.

What it does is:
 - for each dynlib in command line:
    - load dynlibs
    - call the VSTPluginMain function if it exist, 
    - unload it

--------------------- ldvst.cpp --------------------

#include <dlfcn.h>
#include <cstdio>
#include <cstring>
#include <vector>

typedef __cdecl void* (*VSTPluginMain_t)(void*);

int main(int argc, char**argv)
{
    std::vector<char*> dllPaths;
    if (argc < 2)
    {
        printf("usage: ldvst [-lazy] <thing.vst>\n");
        return 1;
    }

    bool lazy = false;

    for (int i = 1; i < argc; ++i)
    {
        char* arg = argv[i];
        if (strcmp(arg, "-lazy") == 0)
            lazy = true;
        else if (strcmp(arg, "-now") == 0)
            lazy = false;
        else
            dllPaths.push_back(arg);
    }

    for (int i = 0; i < dllPaths.size(); ++i)
    {
        char* dllPath = dllPaths[i];

        printf("dlopen(%s)\n", dllPath);
        void* handle = dlopen(dllPath, lazy ? RTLD_LAZY : RTLD_NOW);
        if (handle == NULL)
        {
            printf("error: dlopen of %s failed\n", dllPath);
            return 2;
        }

        VSTPluginMain_t VSTPluginMain = (VSTPluginMain_t) dlsym(handle,
"VSTPluginMain");
        printf("dlsym returned %p\n", (void*)VSTPluginMain);

        if (VSTPluginMain != NULL)
        {
            void* result = VSTPluginMain(NULL);
            printf("VSTPluginMain returned %p\n", result);
        }

        printf("dlclose(%s)\n\n", dllPath);
        dlclose(handle);
    }
    return 0;
}
-----------------------------------------------------

The host is compiled with:

$ clang++ ldvst.cpp -o ldvst-cpp


Here the whole dynlib source:

----------------- distort.d -------------------------

extern(C) void* VSTPluginMain(void* hostCallback)
{
    import core.runtime;
    Runtime.initialize();
    import std.stdio;
    import core.stdc.stdio;
    printf("Hello !\n");
    Runtime.terminate();
    return null;
}

------------------------------------------------------

This dynlib can be built with ldc:

$ ldc2 -shared -oflibdistort.so -g -w -I. distort.d -relocation-model=pic

or with dmd

$ dmd -c -ofdistort.o -debug -g -w -version=Have_distort -I. distort.d -fPIC
$ dmd -oflibdistort.so distort.o -shared -g


For the purpose of demonstration you need another C dynlib, for example 
/System/Library/Frameworks/Cocoa.framework/Cocoa on OS X.


Now the bug triggers when calling:


== How to reproduce ==

$ ldvst-cpp libdistort.so 
   => works

$ ldvst-cpp libdistort.so libdistort.so 
   => works

$ ldvst-cpp /System/Library/Frameworks/Cocoa.framework/Cocoa libdistort.so 
   => works

$ ldvst-cpp libdistort.so /System/Library/Frameworks/Cocoa.framework/Cocoa
   => FAIL, and that's precisely the case that happen in production :(

$ ldvst-cpp /System/Library/Frameworks/Cocoa.framework/Cocoa libdistort.so
/System/Library/Frameworks/Cocoa.framework/Cocoa
   => works

In other words, if the host program loads a D dynlib first, then a C dynlib,
then the second dlopen fail.

This is pernicious since the host program would scan my program, dlclose
successfully, then put in jail another program.

As of today, this is the only thing between me and customers, all other bugs
can be work-arounded, this one I'm not sure.

--