Making an .exe that executes source file inside itself.

[evilrat]

On Thursday, 26 April 2018 at 10:06:57 UTC, JN wrote:
> On Wednesday, 25 April 2018 at 19:19:58 UTC, BoQsc wrote:
>
> Alternatively, I don't know about specifics how to implement it 
> in D, but the key phrase you are looking for is "code hotswap" 
> or "hot loading". It's being popularized right now in gamedev 
> circles, to avoid slow complications of the code. The way to do 
> it is push code into DLLs, then reload the DLL if the change is 
> detected. Of course it's not that simple, because there are 
> some tricky parts around DLL boundaries and you have to reload 
> the entire state, but it is possible. In such case, rather than 
> recompile the entire code, you can just recompile a small 
> subset of functionality to a DLL.

----- content below is just my thoughts and may or may not have 
anything common with reality, don't take it as truth, forget 
after reading, proceed on your own risk
----

The overall process is like this:

1) make a core app that basically serves as host (whether or not 
put domain logic here is open question)
2) when run, scan & plug-in[*] shared libs (they must must follow 
well defined plugin API, ideally providing information about 
their requirements/dependencies and their services)
3) when library should be hot reloaded, it first serializes 
'shared' data, then re-loaded with same mechanism as in 2), and 
deserializes data back.

[*] - it has implications depending on platform, on Windows for 
example don't just load the libs(executing one's is locked by 
OS), but copy to temporary executing location first, that way one 
can overwrite the original libs while (a copy of) it is still 
running and app can be notified about file changes to perform hot 
reload.
(AFAIK UnrealEngine does this without temporary location, instead 
it relies on custom build system that checks whether app is 
running, which will then append incremental number to the name of 
the resulting binary, after build is done editor replaces old 
library with new one. I don't know the exact notification 
mechanism, it could be the build system itself or just FS event 
handled by engine which compares known running lib names for such 
incremental change, both ways makes it harder to manually replace 
library)



The one of the major points of having such libraries, after all, 
to be able to swap implementation without full rebuild, and 
better, in run time.
For example in case of game engine, let's say we want to switch 
rendering API(say DirectX to Vulkan), in this case there is even 
no need to serialize anything because most(if not any) of the 
stuff is implementation specific, such swap could be done 
*almost* in real time, I mean minor freeze for few frames doesn't 
makes a big difference here, and really I doubt anyone would do 
that in a middle of a game(not in menu or pause).

There is still lots of question, after all this is just example 
in theory, in practice however I've never yet come close enough 
to the practical implementation of such thing in D.

---- WARNING: ONLY WINDOWS-SPECIFIC STUFF BELOW

But now the problem.
Guess what? DLL support on Windows is simply isn't there yet! 
AHAHAHAH (imagine some epic evil laugh here)

Sure thing, it is possible to make a bunch of loose functions 
that operates on built-in (value) types, but does this really 
provides the extensibility? I can hardly imagine usefulness of 
such approach in any but trivial cases, just imagine Unity or 
Unreal Engine, or complex software like 3ds Max or Maya...
It is also to some extent possible to go shady with hacks and 
abuses, but this is something that should not be shipped as 
production ready, and there is even no guarantee it will work in 
a future.
The last option probably is to implement whole mechanism from 
scratch, like COM objects, D is a system language after all, but 
I to be honest don't have the desire to fall that low, especially 
because on Linux it works for years...

(side note: I actually tried few simple (hacky) test cases with 
passing classes cross-boundary and then unloading library, seems 
to work on Windows, not sure about memory leaks and consequnces 
in long runs though)