Emulating DLL

[Craig]

On Wednesday, 20 March 2019 at 14:27:54 UTC, Ethan wrote:
> On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:
>> For example, with windows I could simply compile to a dll then 
>> extract the code, or just use LoadLibrary and it effectively 
>> does all the work(steps 1 & 2 & 3).
>
> LoadLibrary performs steps 1 and part of step 2. The DllMain 
> function of a library is called separately by Windows for each 
> thread in your system, which is where the bulk of step 2 is 
> handled. Step 3 isn't handled at all by LoadLibrary, and is 
> instead entirely up to you to deal with via the GetProcAddress 
> function.
>
> If you want DLLs to operate in that step 1-2-3 manner, then the 
> compiler can generate a static library that handles that all 
> for you. But, as you might expect, that removes the hot 
> reloading capability as that is all handled before WinMain is 
> entered.
>
> If you expect hot reloading to work without effort, you're 
> going to have to compile your DLLs with -betterC. The D Runtime 
> is not yet in a standalone DLL (unless I've missed an 
> announcement over the last couple of months). Each DLL you 
> build will require the D runtime to be embedded. Things get 
> really tricky from there.
>
> If you ever use your DLL for more than calling functions with 
> basic types, or value aggregates (structs) containing basic 
> types, then be aware that without -betterC that there is 
> potential for the moduleinfo and typeinfo systems to go a bit 
> screwy. Especially once druntime lives in its own DLL, as 
> druntime expects those systems to be initialised once at 
> startup. POSIX libraries will have the same issues, but I have 
> nod had a need to investigate further over the last few months.
>
>> LoadLibrary is not portable though and seems excessive to do 
>> something that should be rather simple unless there is 
>> something I'm missing that has to be done that is complex.
>
> LoadLibrary is the equivalent to dlopen on POSIX systems, with 
> one very important difference: Windows does not provide lazy 
> symbol evaluation. As such, extra work is required to handle 
> that. The average usecase is covered by the mentioned static 
> lib, but any hot reloading is handled in a custom manner by 
> every Windows codebase you'll encounter.
>
> The static lib I mention, in fact, redirects every cross-DLL 
> call to a jump table. And cross-DLL data sharing gets hairy in 
> Windows. Function parameters aren't a problem, but Windows 
> explicitly disallows thread local variables to cross the DLL 
> boundary for example (even if you expose it for export and 
> perform a symbol lookup, it won't work and you'll get garbage 
> back).
>
> As I said, it's not a simple thing by any means. My work has 
> attempted to hide all that complexity by boiling it all down to 
> "include these headers/modules, link this lib, and get on with 
> life". If you find the resulting code too complex (and I freely 
> admit that I haven't got the integration in to as foolproof a 
> manner as I am aiming for) then very definitely read through 
> the presentations to get a clear idea of why something that 
> should be so simple really isn't.

You are making this more complicated because you don't understand 
that I don't need a general purpose solution. I simply need to 
call simple functions that do not do much, not full blown 
applications.

For example, Suppose I want to write a fractal display 
application and instead of hard coding the function to display, 
it's external.

e.g.,
import std.complex;
alias C = Complex!double;

C Mandelbrot(C z, C c) { return z*z + c; }


This function does not require much of anything to run. The 
machine code can be copied and pasted at will to any executable 
memory location and ran... it can, in fact, be inlined with a 
little caution(reserve enough space and ignore the stack code). 
No initialization needs to be nor garbage collecting.

For a little more robustness one would want to allow for GC, 
which should not be hard, and context passing so more useful 
functionality can be had.


If D had a Dstring2Machine that compiled the above code to 
machine code then it could be used directly without issue.

Because these functions are deterministic and the host determines 
their usage, there is little issues to worry about.  Hot swapping 
is simply changing a pointer when the function is not being 
called.

As long as certain guidelines are followed one can do this quite 
simply without too much issue and it allows for a scripting like 
solution for D apps. It allows one to inject functionality in to 
an application that runs at full speed and is relatively fast to 
compile.

The idea is not to be able to connect two full blow external apps 
but to connect little pieces of functionality to an app that 
allow for vast flexibility. Since each unit of functionality is 
essentially pure, there are no major issues.

It's this simple:

1. Take source and compile it in to machine code.
2. App extracts machine code and inserts it in to it's memory 
space to be able to execute.

Do you agree that, at least in some cases, this is entirely 
feasible and easy?


I'll give you a hint:

1.

module simple;

void foo() { };

compile directly in to machine code(the most raw form possible 
containing foo).

2.

    App load the machine code, execute, reload and execute if 
desired...


Since foo does nothing it's just a nop. We are just copying 
around a ret. It's impossible for anything to ever go awry here 
excluding buggy code/compiler/etc.


 From here one can expand the model to include something that is 
more useful.

The idea is to expand it enough to make it generally quite useful 
but not so much one has to include the kitchen sink.

I don't need a heavy version of binderoo, I need a light one. 
Something that lets me do basic "scripting" so I can modify the 
application behavior at runtime without recompiling the 
application.

If you think the example is stupid and useless then

int age() { return 43; }

and nothing changes.

of course you will then think that example is stupid because we 
could just put it in a text file and read that rather than go 
through all this trouble.

so then

C calc(C z, C c) { return z*z*z*z*z*z + c*z + c; }

and to get around this without leveraging the D compiler one has 
to make a full blown parser... even though it's still just 
relatively basic code like age. It just includes some 
calculations.

It's no different from foo though. Just basic code.

But we can go further by allowing more information to be passed 
to and from the function. The function can also be impure such as

void foo() { writeln("I'm IMPURE!"); }

which will work without much work(might require fixing up a few 
symbols/pointers).


All it is, is delayed compilation. I'm simply wanting to delay 
the compilation of a piece of code in an app from the main 
compilation. All the other stuff about the OS and TLS are 
irrelevant in this context.

One could sort of do this with the object files and just 
recompile the functions and relink everything. But with a little 
work this step can be avoided since it's not needed and 
slower(but more robust).