Embedding D Shared Library in WSGI Web Server

[John McFarlane via Digitalmars-d-learn]

On Tuesday, 28 October 2014 at 09:34:33 UTC, Chris wrote:
> On Tuesday, 28 October 2014 at 00:16:03 UTC, John McFarlane 
> wrote:
>> Hi,
>>
>> I've written a modest shared library in D that I'd like to 
>> call directly from a Python web server (Linux/OS X, Apache, 
>> WSGI, Pyramid). I can call it reliably from within Python unit 
>> tests but on a running server, the library causes a SIGSEGV as 
>> soon as I try anything as complicated as a writeln call. Under 
>> Linux, I've tried using Pyd, CFFI and ctypes to bind the 
>> native .so to Python with the same result. I've tried calling 
>> Runtime.initialize();from an init function on server startup 
>> and from the D function being called as part of the web 
>> request. I've even tried writing a shim in C to make these 
>> calls and call through to the D.
>>
>> I've compiled with DMD and GDC with a variety of switches. 
>> Currently my build command looks like:
>>
>>    dmd ./d/*.d -version=library -release -shared -O -inline 
>> -defaultlib=libphobos2.so -fPIC -oflibgusteau.so
>>
>> The extern(C) function takes a string and returns a string. 
>> I've tried holding onto a pointer to the returned string and 
>> passing in an adequately sized byte array from Python. Smaller 
>> strings seem to cause a crash with lower probability but 
>> typically the input and output strings are a few KB in size 
>> and consistently crash. I taken measures to ensure that they 
>> are correctly encoded and null terminated. I've also tried 
>> disabling the GC, calling terminate on function exit (after 
>> copying result into received buffer) and various other 
>> measures in an attempt to get something working.
>>
>> I'm guessing that the web server spawns/relies on a thread or 
>> process for each request and I've read that runtime 
>> initialization should be invoked from the main thread. Does 
>> this always mean the first thread that was created on process 
>> start up or does it just have to be consistent? If calling 
>> from a thread other than the one used to initialize, is that a 
>> problem? Are other threads created when GC is invoked which 
>> might last past the extern function being called and if so, 
>> can I prevent this by controlling collection explicitly?
>>
>> Thanks, John
>
> I had a similar problem with a DLL for Python. The reason my 
> DLL would crash were some writeln statements. After removing 
> them it worked fine. I suppose the writeln messes things up, be 
> it because of threads or because of conflicts in the file 
> system (remember writeln is a file system operation). I suggest 
> you have your lib not print things to console, but use a 
> different mechanism instead, e.g. pass a string back to Python 
> and have Python do the printing (if needs be).
>
> You could also try to set your lib up as a small socket server 
> that waits for input and answers. That usually works for me.

Thanks for the suggestion. I looked into removing all of my 
writeln calls and
found the I still had the same problem. Simply a call to 
to!string with the
input char* is enough to cause a crash. I will go with the socket 
server as a
last resort but it would be much easier (especially from a 
testing POV) if I
could call the native D code directly from the web server.