D scripting in D

[Lewis via Digitalmars-d-learn]

On Friday, 2 June 2017 at 20:47:31 UTC, Mike B Johnson wrote:
> Would you mind, when you get some time, to write up a more 
> detailed analysis of the problems you had to overcome to get it 
> to work? Possibly we could get some type of library solution 
> that just "works" with very little change and restriction?

For sure. I actually want to post the source code at some point, 
but the changes I made are very much set up specifically for this 
project. I'll sift through at some point and see if I can gather 
together something worth posting.

The main problem with hot-reloading DLLs stems from the fact that 
every new D DLL comes with its own copy of druntime. This means a 
second GC, a second registry of active threads, a second registry 
of typeinfos, and so on. If there were a way to load a D DLL that 
automatically used the executable's copy of druntime, most of the 
hacks I had to make could be removed.

For reference, my general approach is that I have an EXE and a 
DLL. We run the EXE, which makes a copy of the DLL and loads the 
copy. Once per frame, the EXE calls GameUpdateAndRender() on the 
DLL, which does everything needed to step the game forward one 
frame. After each call to GameUpdateAndRender(), the EXE checks 
to see if the original DLL has changed. If so, we unload the DLL, 
make a copy of the new one, and load the copy. We then continue 
calling GameUpdateAndRender() as usual. All game state that needs 
to survive a hot reload is stored directly (or referenced 
indirectly) from a giant struct call the GameState. The EXE keeps 
a void* to the GameState, that way the GC doesn't collect 
anything inside it.

The main issues I remember off the top of my head:

- I have to use the GC proxy to redirect DLL GC calls over to the 
EXE's GC. This is because when I hot reload, the DLL's GC dies, 
so it can't be holding on to anything that needs to survive the 
hot reload (and I want pretty much everything to survive a hot 
reload).

- D has some built-in support for redirecting GC calls. However 
when you load a DLL, by the time you get a chance to call 
gc_setProxy(), it's too late, and druntime has already done a few 
allocations into the DLL GC. I had to change the order of a 
couple parts of initialization (in runtime.d I think?), and in my 
own code defer calling dll_attach_process() until after 
DllMain(). That way I get a chance to set up the GC proxy, then 
initialize druntime, which will trigger a few allocations that 
now go to the right place.

- I haven't experimented thoroughly with threading, but I think I 
have to do something similar. Since we're using the EXE's GC, if 
the EXE's druntime doesn't know about the existence of a thread 
spawned through the DLL's druntime, then we run into problems. So 
I redirect all thread spawning over to the EXE.

- With threading, we have to be careful that we don't tear down 
the DLL from underneath a thread that might still be running code 
on it. We have two options. 1. Join all threads before unloading 
the DLL.
2. The thread needs to be running a procedure that originates in 
the EXE, and then calls into the DLL, returning periodically back 
to the EXE. While execution is back in the EXE, we take that 
moment to reload the DLL.

Currently I'm doing the former, but might move to the latter if I 
end up needing long-lived threads for performance reasons. Note 
that callbacks originating from C code running on a separate 
thread also have this same restriction, and need to be handled 
careful.

- The different druntimes have different typeinfos for each 
class. This can be problematic for a few reasons, and I don't 
have as total a grasp on what situations related to classes are 
guaranteed to cause a crash. In particular, finalizers seem to 
cause problems, so I have disabled them entirely in druntime. In 
my own code, I almost exclusively use structs, and any instances 
of classes I do use are never stored long-term, and are never 
allowed to survive a hot reload.

I think those are all the D-specific gotchas, but I'll 
double-check my notes when I get home. In addition, there are 
common gotchas to this approach that would exist in other 
languages (C/C++ for example). Examples:

- Static variables will die in a hot reload, so be cautious using 
them.
- Storing a char* or string that references string in the data 
segment of the DLL will cause problems when we hot reload. To 
avoid this, I have a templated function that, on hot reload, 
iterates over my entire gamestate and copies each string. Since 
hot reloading is a debug-only utility for me, this is acceptable.
- Code changes to the gamestate struct will require a restart. 
Adding a new field to the gamestate does not, since I have some 
padding room at the end of the struct where I can add new fields 
during a hot reload.
- Don't hold on to function pointers through a hot reload, as 
functions addresses are liable to be reshuffled in the new DLL.
- Probably some others I'm forgetting.

In terms of making a library solution, I'm not sure what the best 
approach would be. But allowing users to compile a "light" D DLL 
that auto-patches itself back to the calling EXE's druntime would 
drastically reduce the number of hacks needed to make this work. 
So I guess that would be the main hurdle to overcome?