threading issues with D -> C -> Python

[Ellery Newcomer via Digitalmars-d-learn]

On 12/04/2014 02:11 PM, Michael wrote:
> On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote:
>>
>> dustmite?
>
> Not sure what went wrong with dustmite, but every time I tried it it
> just started deleting all the files in the directory and setup.py would
> give errors. I manually deleted a reasonable chunk of the code and I'm
> left with these files which still seem to cause segfaults:
>
> Main code: http://pastebin.com/zqgNTk9w
> PyD definitions: http://pastebin.com/6mRH3KZZ
> setup.py: http://pastebin.com/i9Ph78UC
> test code that causes segfaults: http://pastebin.com/1ukzShVh
>
> Cheers,
> Michael.

hmm.. looks like here it originates in python when it tries to acquire 
the GIL. specifically, pthread_cond_timedwait is segfaulting.

in your code, execution inside a python thread makes it to 
receiveTimeout in get_image. it made it past receiveTimeout in acquire.

then I guess there is a context switch. the main python thread throws an 
exception, but a number of things trigger the segfault. I think it's 
just the interpreter loop calling RestoreThread. backtrace looks like

#0  pthread_cond_timedwait@@GLIBC_2.3.2 ()
     at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000003799d07bb3 in PyCOND_TIMEDWAIT (cond=0x379a063220 <gil_cond>,
     mut=0x379a0631e0 <gil_mutex>, us=5000)
     at /usr/src/debug/Python-3.3.2/Python/condvar.h:103
#2  take_gil (tstate=tstate at entry=0x604410)
     at /usr/src/debug/Python-3.3.2/Python/ceval_gil.h:224
#3  0x0000003799d081fb in PyEval_RestoreThread 
(tstate=tstate at entry=0x604410)
...

It looks like this is the python main thread.

I see two other threads. (i took out one of your python spawns) #2 looks 
to be your listener thread. std.concurrency.send seems to have gotten it 
into a gc_malloc, but it looks like it's just waiting. it's currently in 
sem_wait. this one would have been spawned in D code by #3

#3 is your other python thread. it is also in pthread_cond_timedwait. by 
its stack trace, receiveTimeout is just waiting.


I guess tomorrow I can try messing around with thread_attachThis, as the 
fullcollect happening in #2 might be screwing with python data. But you 
aren't really passing anything from python to d or vice versa, so I'm 
not sure why the gc would need to know about the python threads.

not exactly my area of expertise, this.