Druntime and non-D threads

[Joakim]

On Friday, 8 December 2017 at 09:33:03 UTC, Ali Çehreli wrote:
> I'm trying to use D as a library to be called from a non-D 
> environment e.g. Java runtime. If I'm not mistaken, it's quite 
> difficult and perhaps impossible to use GC in such a scenario. 
> It works as long as attached threads don't go away either by 
> themselves or by thread_detachThis.

I've been doing this for some time, by running all the D stdlib 
tests in a shared library that's called from Android's Java 
runtime, no problem with the GC or threads, if I set it up right 
and with a tweak or two:

https://wiki.dlang.org/Build_D_for_Android#Changes_for_Android

However, I go the other way and call Java methods from D, so it 
does depend on whether the process running the D shared library 
is long-running or not, as I've had issues when a D function or 
two are called periodically from a Java app instead.

> My setup is Linux (Ubuntu-based), dmd 2.077.1, 64-bit build. D 
> is used in a shared library that is called by non-D threads. 
> (Tested with C and Java.)
>
> 1) The following newsgroup topic is about calling 
> thread_attachThis() for threads created outside of D:
>
>   http://forum.dlang.org/post/ounui4$171a$1@digitalmars.com
>
> As suggested in that thread, I think I have to call 
> thread_detachThis but I'm not sure when that can be safely 
> done. One idea was to attach and detach in every api function 
> something to the effect of
>
> extern(C) my_api_func() {
>     thread_attachThis();
>     scope(exit) thread_detachThis();
>
>     // Do work, potentially producing garbage...
> }
>
> Does that make sense? Wouldn't garbage produced by that thread 
> leaked after detaching? However, failing to detach would be bad 
> as well as the calling thread can terminate without our 
> knowledge. (More on that below.)
>
> 2) Obviously, Runtime.initialize() must be called for Druntime 
> to work at all. Question: Is the thread that calls 
> Runtime.initialize() special compared to the other threads? Can 
> this thread disappear and the Druntime still work?
>
> 3) An attached non-D thread can exit without any notice 
> (gracefully or otherwise) while it's still attached to D's GC, 
> causing segmentation faults or deadlock.
>
> I failed to find a way for Druntime to be resilient when such 
> threads disappear. For example, the registered cleanup handler 
> in thread.d is called only for cancelled threads, not the ones 
> that exit simply by returning from their thread functions. 
> (This is according to cleanup handler spec.)

I haven't had to try all these thread registration methods, 
perhaps because the apps I'm testing are much simpler or because 
I'm going the other way from D to Java most of the time.

> 4) Druntime uses pthread_kill to signal threads to suspend (and 
> resume) threads. However, successful return of this function 
> does not mean that the thread will respond to that signal. So, 
> we have a couple of bugs in Druntime as the number of 
> sem_wait() calls we make depends on the unreliable return value 
> of pthread_kill. Perhaps that's the reason for bugs like the 
> following:
>
>   https://issues.dlang.org/show_bug.cgi?id=15939
>
> I don't see a way out of this POSIX limitation. 
> (pthread_key_create may help as a "thread destructor" but I 
> haven't played with it yet. thread.d beat me up pretty bad for 
> more than two days; I'm too tired to do anything else right 
> now. :) )
>
> 5) We depend on SIGUSR1 (and SIGUSR2, which may not be 
> necessary but it's a different topic) to suspend non-D threads. 
> Does that work with all threads? What if the calling framework 
> has other uses for those signals? Would we be interfering with 
> them?

Those signals are used for D threads, should work fine unless 
they're being intercepted somewhere, as they are by the Android 
runtime.  However, you can always change the signals used, as I 
did by swapping them on Android, and as others are trying to for 
other reasons:

https://github.com/dlang/druntime/pull/1851#discussion_r123886260
https://github.com/dlang/druntime/pull/1565

> So, what are the rules of using D as a library for a non-D 
> framework? I have the following so far but I'm not sure on all 
> points:
>
> - SURE: One thread must make a call to Runtime.initialize()
>
> - SURE: Every D api call must call thread_attachThis
>
> - SURE: Attached threads must *not* terminate gracefully, due 
> to error, or by cancellation. (As there is no way of 
> guaranteeing this in POSIX, I think using D as a library in a 
> framework is best-effort at best.)
>
> - NOT SURE: thread_detachThis must *not* be called as the 
> thread may have uncollected garbage.
>
> - NOT SURE: SIGUSR1 and SIGUSR2 should be available.

I have tried to avoid all these problems by having the D shared 
library be the starting point of the app and calling Java 
functions occasionally instead, so haven't delved into all this.