threading issues with D -> C -> Python

Tue Dec 2 22:29:53 PST 2014

On Wed, 2014-12-03 at 01:07 +0000, Michael via Digitalmars-d-learn
wrote:
> Hi. I'm new here and this is my first post. I'm not sure this is 
> the right subforum for it, but wasn't sure where else to put it 
> either.
> 
> I've written a library to talk to some external hardware using a 
> socket. It uses the std.concurrency threads to send messages 
> between the main D-object for the hardware and the D-object for 
> the sockets. I then wanted to be able to call these functions 
> from Python. PyD appeared to be out of date, so I've been using a 

As far as I can tell PyD is still active, but in a non-funded FOSS way,
i.e. work happens as and when volunteers put time and effort in. I
haven't tried PyD recently but it worked fine last time I did. If can
set out what you tried and what didn't work, maybe there is a PyD
solution, or a fix to PyD to give a solution?

> D -> C interface, and a C -> Python interface. The python code 
> will often run from different python threads, so I then added yet 
> another message-passing layer between the D->C interface and the 
> D->hardware interface.

D's "big problem" is shared objects/dynamic link libraries. Without them
you cannot interwork with Python at all. I have tried experiments on
Linux creating shared libraries from D code with C linkage entry points
to create classic Python extensions, and it appears to work fine. Except
for having to start up the D heap and thread  management, should they be
needed. But that is what PyD is there for. If I took my experiments any
further I would end up recreating PyD or something like it.

It sounds like you are in a similar situation except that you appear to
have an extra layer of C code. I am not sure a layer of C is needed
between Python and D, it would be good to know more about why you seem
to need it.

Your use case is interesting as I have more or less given up on using D
in a Python context. CPython naturally requires C linkage shared objects
for non-Python code so C, C++ or D would be fine except that everyone
does it in C or C++, very few people in the arena have even heard of D.
PyPy brings it's own issues with non-Python code but the now have a C
capability. With Cython, Pythran, and more recently Numba there is
increasing less and less need for any user written non-Python code.
Hardware control would be done in C and PyPy now has a C linkage
mechanism.

Of course for Python networking there is Twisted, Asyncio and Tornado so
no Python folk are using non-Python code for handling networking.

> My problem is that this code routinely causes segmentation 
> faults. I've spent a long time going through trying to figure out 
> exactly what the causes are. I think there have been some related 
> to D-exceptions not being handled gracefully by the C/Python 
> code. Some more by stdout writing from multiple threads (which 
> surprised me).
> 
> I'm fairly sure I have tackled both of these issues, but it still 
> seems like Python threads and D threads don't mix well. When 
> running the same functions from D, I am able to get no errors, 
> but when run from Python/C it causes segfaults reliably.

Without seeing your code, it is difficult to say what may or may not be
the problem, but I would guess it is about D infrastructure start up. I
recollect being able to reliably get segfaults this way.

Are you using ctypes or CFFI on the Python side?

> Sorry for the large exposition. I am currently at the point of 
> suspecting bugs in Phobos, but I am unskilled enough to tell for 
> sure, and would appreciate any help.
> 
> The latest core dump gives a backtrace of almost entirely phobos 
> commands:
> 
> #0  0x00007fe789ad3b97 in gc.gc.Gcx.fullcollect() () from 
> /lib/libphobos2.so.0.66
> #1  0x00007fe789ad3294 in gc.gc.Gcx.bigAlloc() () from 
> /lib/libphobos2.so.0.66
> #2  0x00007fe789ad0df1 in gc.gc.GC.mallocNoSync() () from 
> /lib/libphobos2.so.0.66
> #3  0x00007fe789ad0c15 in gc.gc.GC.malloc() () from 
> /lib/libphobos2.so.0.66
> #4  0x00007fe789ad6470 in gc_malloc () from 
> /lib/libphobos2.so.0.66
> #5  0x00007fe789ae6d36 in _d_newitemT () from 
> /lib/libphobos2.so.0.66
> #6  0x00007fe789e57112 in 
> std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from 
> /usr/lib/libv5camera.so
> #7  0x00007fe789e570b5 in 
> std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from 
> /usr/lib/libv5camera.so
> #8  0x00007fe789e562dc in 
> std.array.__T8AppenderTAaZ.Appender.__T3putTAaZ.put() () from 
> /usr/lib/libv5camera.so
> #9  0x00007fe789e561ea in 
> std.array.__T8AppenderTAaZ.Appender.__T3putTxwZ.put() () from 
> /usr/lib/libv5camera.so
> #10 0x00007fe789e5617d in 
> std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() 
> () from /usr/lib/libv5camera.so
> #11 0x00007fe789e56132 in 
> std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() 
> () from /usr/lib/libv5camera.so
> #12 0x00007fe789e61f09 in 
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
> () from /usr/lib/libv5camera.so
> #13 0x00007fe789e5b4ac in 
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get() 
> () from /usr/lib/libv5camera.so
> #14 0x00007fe789e57e8d in 
> std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T6__ctorTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.__ctor() 
> ()
>     from /usr/lib/libv5camera.so
> #15 0x00007fe789e581f1 in 
> std.variant.__T8VariantNVmi32Z.VariantN.__T7handlerTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.handler() 
> ()
>     from /usr/lib/libv5camera.so
> #16 0x00007fe789e57d0f in 
> std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T8opEqualsTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.opEquals() 
> ()
>     from /usr/lib/libv5camera.so
> #17 0x00007fe789e57ba8 in 
> std.typecons.__T5TupleTAyaTiTG65536kZ.injectNamedFields() () from 
> /usr/lib/libv5camera.so
> #18 0x00007fe789e62087 in 
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
> () from /usr/lib/libv5camera.so
> #19 0x00007fe789e621a3 in 
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
> () from /usr/lib/libv5camera.so
> #20 0x00007fe789e5b7f6 in 
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get() 
> () from /usr/lib/libv5camera.so
> #21 0x00007fe789ac7d51 in core.thread.Thread.run() () from 
> /lib/libphobos2.so.0.66
> #22 0x00007fe789ac6f95 in thread_entryPoint () from 
> /lib/libphobos2.so.0.66
> #23 0x00007fe79cee5182 in start_thread (arg=0x7fe77aca5700) at 
> pthread_create.c:312
> #24 0x00007fe79cc11fbd in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

My guess would be not properly initializing the D infrastructure from
the incoming Python thread.

I would suggest that you want to avoid threads crossing the boundaries
and just pass data via a shared channel. Unix pipes seem to work well in
this context since they provide a language independent data channel.  

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://lists.puremagic.com/pipermail/digitalmars-d-learn/attachments/20141203/20d3eb02/attachment.sig>