Invoking the compiler during runtime

[H. S. Teoh]

On Wed, Aug 05, 2020 at 06:02:58AM +0000, cy via Digitalmars-d-learn wrote:
> D's compile-time-execution is fantastic, but there are some times when
> I'd like to examine the generated code, or produce code that needs to
> pass through earlier phases before CTFE, or do AST stuff. Sometimes I
> simply don't want to generate the code with every compilation, so
> saving the generated code in a file some would be really neat, if I
> could then invoke the compiler during runtime to build it. Plus
> backtraces wouldn't just be "dlang-mixin-397."

TBH, in spite of D's amazing metaprogramming capabilities, there comes a
point where compile-time generated code just becomes too unwieldy. Like
if you have multi-page mixin strings, or mixins nested to the n'th
level, or your token strings span 10+ pages and it becomes just
impossible to debug.

In those cases, my standard solution is to write an auxiliary program
that emits D code, and then just compile that with the main program as a
second step. The nice thing about that is, as you said, you can examine
the generated code, run coverage on it or whatever other analysis, or
just plain debug the generating code until the output looks like it
might compile before you actually try to compile it, instead of dealing
with multi-page template instantiation errors that only a Klingon could
decipher.


[...]
> In C I write a utility program in awful C to print out C code, have a
> cmake custom command run that awful C as needed, and assume the
> existence of that generated source in my main program.

That's exactly what I'd do.  The utility program, of course, would also
be in D. :-D  With unittests, even, that ensure the output is
syntactically correct. :-P  How's that for self-reference? :-P


> So is that the best way to do it? Have a side program that writes D
> source to a file, and then the main program simply imports the file?

Yep.


> Or is there some AST syntax I should be generating instead? Some way
> to import the compiler itself, instead of calling it in a subprocess?

IMO it's not worth the trouble to import the compiler itself, unless you
want to do runtime dynamic (re)compilation.  I've also done that, but
generally I advise against importing the compiler, but instead just run
it as a separate process (say using std.process, which is very
convenient).  Generate the code you want to compile in string form, spit
it to a temporary file and compile it (or pipe it through stdin to `dmd
-`) into a shared object, then load it dl_open on Posix (or whatever the
Windows equivalent it) and run the code that way.  Before the recent
fiasco, dmd is fast enough that generally you won't notice the lag
unless you do this in a tight loop.

But I wouldn't recommend this unless you want to do it at runtime, since
you'll have to ship a copy of dmd with your program, which comes with
its own bag o' worms (you essentially need to ship an entire working
installation of dmd for this to work).  If you just need to generate
some complicated D code at build time, just write a helper utility to
emit D code, and call it a day.

I have a project where I need to generate D code from large data files
that would be too resource-intensive to do from inside CTFE, so a
utility that emits D is what I went with.

In the same project I also have a helper program that scans GLSL (vertex
shader) code and generates D APIs for each shader (automates binding
parameters, generation of wrapper functions, etc.).  Possible to do with
CTFE if I tried hard enough, but at the end of the day, why would I?
Doing it in CTFE bloats compile-time beyond my patience, and requires
onerous resources, and besides, CTFE cannot access OS functions like
scanning directories; so why not just write a helper D program that has
full OS access, runs faster, and does what it needs to do to generate D
code that I can then import in my main program.


> Is there a way in dub to specify that you run a D program contained in
> X.d during the build process, to build Y.d, that Z.d imports?

I don't know if the latest version of dub can do that, but the last time
I checked about a year or so ago, it couldn't. Or at least, not without
jumping through hoops and bending over backwards.  But that's not a
problem for me anyway, since I use SCons to build my projects, and SCons
has no problem at all building multiple helper programs that each
generate some subset of D source files, and then compiling said files
into the final product. In fact, even multiple final products (in the
aforementioned project, for example, my SCons script builds both an APK
for Android with an LDC cross-compiler, and a Linux executable for on-PC
testing at the same time with dmd :-P).  This method of building
generally goes against the grain of dub's design, though, so instead of
fighting with dub in an uphill battle, I just went with something more
flexible instead.


T

-- 
Windows 95 was a joke, and Windows 98 was the punchline.