Do we already have full compile time / runtime separation?

[Jonathan M Davis]

On Thursday 23 December 2010 01:11:40 Mariusz Gliwiński wrote:
> Hello,
> I've been trying to manage this on my own for like 2 days but still
> couldn't do that, and because my brain just suddenly turned-off, I would
> ask You for some guidance.
> 
> The thing is:
> I'd like to make some kind of messaging in my application. So, there is
> - interface Msg
> - classes that are implementing it, i.e. MsgWindowClose
> - interface Provider for message publishers, i.e. MsgProviderWindowClose
> - interface Listener for message subscibers i.e. MsgListenerWindowClose
> - interface Handler for message filters i.e. bool
> MsgHandlerWindowClose.log() - interface Mediator for message passing
> between program classes, i.e. MsgMediatorDMultiThreading() - which is
> using D messages btw.
> 
> Nothing related to D so far. Key thing is, that most of this Msg objects
> will be really generic, so I'm building object generator, i.e
> MsgProviderGen!"WindowClose":
> <code>
> template MsgProviderGen(string MSG, Args...) {
> 
> 	const char[] MsgProviderGen = "class MsgProvider"~MSG~" : MsgProvider {"
> 
> 			~"override Msg"~MSG~" getMsg() {"
> 				~"return new Msg"~MSG~";"
> 			~"}"
> 
> 		~"}";
> 
> }
> </code>
> Which will be bigger of course.
> 
> Then, for standard and generic messages i can easily define:
> <code>
> private import
> 	msg.msg,
> 	msg.provider.gen,
> 	msg.handler.gen;
> 
> class MsgWindowClose : Msg {
> 
> }
> mixin(MsgProviderGen!"WindowClose");
> </code>
> 
> 
> So far so good, but then if I'd like to add
> <code>mixin("immutable uint id="~static_id~";")</code>
> for each *class* (not object) I got a
> 
> PROBLEM:
> Each compile-time variable has to be const, right? So I can't increment my
> `static_id` each time I build a class (how many times template has
> generated provider/listener of this type). This wont let me statically
> check if each listener has its provider.
> 
> Surely it's not the only use of this feature, For loosen coupling, I'd wish
> to add function that statically returns array of Msg defined in module. It
> isn't an option without compile-time variables too.
> 
> Is it something about undefined module initialization?
> Or maybe there is any way to overcome this problems, since I'm still new @
> D language.
> 
> Ps. How to make associative array of dynamic arrays?
> <code>MsgHandler[hash_t][] _handlers = new MsgHandler[hash_t][];</code>
> wont work.

You're mixing up several concepts, which complicates things a fair bit.

When directly initializing static variables or member variables, the values used 
to initialize those variables must not depend on ordering. That generally means 
that the values used to initialize such variables must either come from 
constants, templates, or CTFE. So,

auto a = 7;
auto b = a;

is not legal, because a is not constant. If it were

immutable a = 7;
auto b = a;

then it's legal. However, note that b itself is not const, immutable, or an 
enum. It's fully mutable. Other ways to initialize such variables would be 
through eponymous templates and calling functions via CTFE (compile-time 
function evaluation):

template add(int a, int b)
{
    enum add = a + b;
}

double multiply(double a, double b)
{
    return a * b;
}

auto a = add!(7, 5);
auto b = multiply(2, 3);

Enforcing the lack of ordering in direct initialization for static variables and 
member variables makes it so that you avoid bugs like you get in some languages 
when you try to do something like

auto a = b;
auto b = 7;

and a ends up being garbage because the variables are being initialized in order 
and b hasn't been initialized yet. It also makes it potentially much faster to 
compile static and member variables, because they can actually be compiled in 
parallel instead of enforcing an ordering to them. And, of course, if you did 
something like

auto a = b;
auto b = a;

the circular dependency would screw you. If you have such ordering issues or if 
you need such variables to be mutable and yet able be initialized from one 
another, you need to use static constructors (which also work on immutable 
variables as long as you initialize them only once, though enums must still be 
known at compile time, so it doesn't work for them).


Now, for templates, the template itself depends on the values/types of its 
parameters. It's pure code generation. A classic example would be something like

struct Pair(T, U)
{
    T first;
    U second;
}

When you declare Pair!(int, float), the compiler creates code similar to

struct Pair!(int, float)
{
    int first;
    float second;
}

If you declared Pair!(double, double), then you'd get a second Pair struct type 
generated. Because code is being generated like this, the template parameters - 
be they types or values - must be known at compile time. So, the rules for what 
can be passed as a template argument are essentially the same as what can be 
used to initialize a static or member variable directly.


Now, with string mixins, you are also generating code. That code must obviously 
be known at compile time. And any values used in them must be known in the same 
way that they are for template parameters and the direct initialization of 
static variables and member variables. So,

mixin("int " ~ varName ~ " = " ~ varValue ~ ";");

would require that varName and varValue be known at compile time. It could be

mixin("int " ~ genVarName() ~ " = " ~ varValue!(5, 2, "hello") ~ ";");

instead (assuming that the appropriate function and template are defined), but 
the values still have to be known at compile time.


Now, as for your problem. If I understand correctly, you are looking to generate 
some set of classes, and have each generated class have a unique ID. You want to 
do this by having a counter which is incremented each time that you have create 
such a class. Well, you _can_ do it but not sanely. The problem, of course, is 
that you can't _use_ a mutable variable at compile time unless its a local 
variable in a function used with CTFE (or a member variable of a struct used in 
a function in CTFE). You can _declare_ them, but you can't use them. And for 
what you're doing, you need to use them. Unfortunately, at the moment I can only 
think of one possible solution, and I'm not sure that it's currently possible.

Generate a UUID for each class ID at compile time. The ID will then be unique 
without the need for each counter. You'd simply call a function with CTFE which 
generated a unique ID. Now, I'm not sure if you can make C calls with CTFE or 
not (I'm guessing not, but you might), and I doubt that anyone has written a 
UUID-generating library for D just yet. I'm also not entirely sure how UUID-
generation works, and it may require internal, mutable static variables, which 
would make this permanently impossible. So, I'm not sure that that's an option 
at the moment, or if it ever will be.


Allowing you to have mutable static variables useable at compile-time would be a 
serious hindrance to the language in general, complicating variable 
initializations considerably, and slow down compile times - potentially by a 
lot. So, I believe that in the general case, D made the correct decision here. 
That being said, it would be nice to have a solution for the sort of situation 
that you're currently dealing with. Unfortunately, I can't really think of one 
at the moment. Everything that I can think of would either outright not work or 
has some flaw in it which makes it so that it almost works but not quite.

Now, if you're willing to forgoe having the IDs being known at compile time, 
there _is_ a way. What you do is you initialize them in a static constructor. 
For example,

class A
{
    static this()
    {
        ID = globalIDCount++;
    }

    static immutable uint ID;
}

That way, the ID is unique, and it's still immutable. Now, this does have a 
potential problem. Normally, global and static variables are thread-local. But 
this in an immutable variable, and immutable globals or statics generally are 
treated as shared by the compiler. In fact, there is currently an open bug on 
the fact that immutable variables can be re-initialized because a static 
constructor was run in multiple threads. How this will be fixed, I don't know. It 
may end up requiring immutable globals and statics to be initialized in shared 
static constructors, or it may require that you have immutable static 
constructors (similar to how you can have immutable constructors) which then 
initialize all immutable globals and statics. Until then, however, you risk re-
initializing ID every time that a thread is created.

To fix this, you can make the static constructor and ID shared.

class A
{
    shared static this()
    {
        ID = globalIDCount++;
    }

    shared static immutable uint ID;
}

but that means that you then have to worry about synchronization among threads. 
You don't want two shared constructors accessing globalIDCount at the same time 
(I'm 99% sure that that can't happen, since all share, static constructors 
should run once on program startup and all run in the same thread, but I'm not 
100% certain that it's guaranteed that only one thread is used or that it's 
required by the language). But using a synchronized block on globalIDCount 
should solve that problem, I believe (I haven't used synchronized blocks much 
though, so this might not be quite correct):

class A
{
    shared static this()
    {
        synchronized(globalIDCount)
        {
            ID = globalIDCount++;
        }
    }

    shared static immutable uint ID;
}

In any case, by using static constructors and delaying the initialization of the 
IDs until runtime, you _should_ be able to get it to work. But you won't be able 
to do it at compile time like you've been trying to do.

- Jonathan M Davis