Executing pure D at compile-time

[kris]

Following on from the "Regex Redux thread, it seems to me there's an 
easy way to execute pure D at compile-time. A few elements are needed:

1) the ability to describe a compile-time function call
2) the facility to pass arguments to it, and recieve a return value
3) a means of identifiying the D code to execute
4) a manner in which the pure D is executed
5) a mechanism for ensuring the executed code is docile

What follows is purely an illustration, since there are a number of ways 
to achieve the same result:

1) Making the call. let's assume standard calling syntax is enabled. 
Perhaps something like this (at the call site):

#  char[] result = regex ("[0-9]", "abc123");

That may not be an entirely practical syntax, but it hopefully gets the 
idea across?


2) Argument passing. The D source file is text, so a simple 
implementation might pass text-arguments to the compile-time function. 
Given that a mixin() is text-based also, it might make sense for the 
function to return text too e.g.

# char[] regex (char[] pattern, char[] string) {}

Note that this function is composed of nothing but standard D code. The 
args are represented by strings for the sake of simplicity; but it could 
also be something more sophisticated.


3) Function identity. The compiler would need to distinguish between 
compile-time functions and all other code (so that it knows what is 
what). One way to do this is to introduce a variation upon public / 
private / package, called 'extension':

# extension char[] regex (char[] pattern, char[] string) {}

With the 'extension' keyword, the compiler can identify 'regex' as a 
compile-time function. Thus, the regex call noted earlier would be 
evaluated as a /compile-time invocation/ of the regex function; as 
opposed to a runtime call. Note that these 'extension' functions would 
be omitted from the target binary: they are for compile-time use only.


4) Execution. One could write a D interpreter and embed it in the 
compiler, but that's perhaps a bit impractical. Instead, why not simply 
recurse the compiler (or spawn a child instance) to generate a seperate 
binary instance of the compile-time function?

Under Win32, for example, the binary could be generated as a .exe file, 
and be passed arguments as normal. The return of the function could be 
captured via an stdout pipe. Better, a dll could be generated instead, 
and be dynamically bound to the executing compiler. The latter has 
several benefits, the most obvious being raw throughput.


5) The problem with enabling pure D at compile-time is a catch-22. You 
want the expressive power and raw execution speed, but you want to 
ensure it doesn't do anything bad. This is a problem regardless of how 
#4 is implemented. However, I rather suspect the OS will provide the 
answer for such concerns? I mean, doesn't Vista (for example) provide an 
execution 'sandbox' where the target is not permitted to create any 
handles? Without handles, there's no file, socket or registry access. 
That's a nice sandbox.


How about an illustrative example? The regex discussed previously?

========
module main;

import regex;

void main()
{
     // result is generated at compile-time ...
     char[] result = regex ("[0-9]", "abc123");
     writefln (result);
}

-----

module regex;

import std.regexp;

extension char[] regex (char[] pattern, char[] string)
{
      auto exp = new RegExp (args[0]);
      return exp.find (args[1]);
}

=========

Note that import operates here exactly as it does today. As does 
everything else. The distinction is the introduction of an 'extension' 
keyword, and the mechanism to invoke the described function at 
compile-time from a call-site (rather than generating a runtime call).

All told, the various posts on compile-time functionality are really all 
about compiler extensions. The degree of extension is just different 
across posts. Supporting a pure D approach is certainly better than 
inventing another language inside D itself; is it not?

Taking this a little further, there's no need for the 'extension' code 
to be generated for each invocation. It can easily be cached by the 
compiler at runtime; particularly a dll implementation. Indeed, assuming 
the sandbox is in place, there's nothing to prevent one from using 
pre-compiled extensions instead:

==========
module regex;

extension char[] regex (char[] pattern, char[] string);
==========

In this case, the extension is simply /declared/ like an extern D 
function would normally be. (there's a assumption that the dll name 
would be somehow bound to the function name. And, of course, the 
assumption that one can sandbox).

The beauty of this approach is in the simplicity and the power. Occam's 
Razor would appear to be at work.

Thoughts?

- Kris