Whole source-tree statefull preprocessing, notion of a whole program

[Vladimir Panteleev via Digitalmars-d]

On Saturday, 8 April 2017 at 10:11:11 UTC, Boris-Barboris wrote:
> 1). I had to save template parameter group_name_par into 
> group_name. Looks like template mixin doesn't support closures. 
> A minor inconvenience, I would say, and it's not what I would 
> like to talk about.

Template mixins' scope is the expansion scope, not the 
declaration scope. (This is useful in some situations, but 
recently I have been mostly avoiding template mixins.)

> 2). After preprocessing I wish to have fully-typed, safe and 
> fast Config class, that contains all the groups I defined for 
> it in it's body, and not as references. I don't want pointer 
> lookup during runtime to get some field.

Looks like your current implementation does not go in that 
direction, seeing as it uses properties for field access.

For such tasks, I would suggest to split the representations 
(native data, DOM tree, JSON strings etc.) from the 
transformations (serialization and parsing/emitting JSON). E.g. 
your example could be represented as:

struct Config
{
     struct TestGroup
     {
         string some_string_option;
         double some_double_option;
     }
     TestGroup testGroup;
}

Then, separate code for serializing/deserializing this to/from a 
DOM or directly to/from JSON.

Individual components' configuration can be delegated to their 
components; their modules could contain public struct definitions 
that you can add to the global Config struct, which describes the 
configuration of the entire application. I've used this pattern 
successfully in some projects, incl. Digger: 
https://github.com/CyberShadow/Digger/blob/master/config.d#L31-L36

>     2.2) Sweet dream of 2.1 is met with absence of tools to 
> create and manipulate state during preprocessing. For example:

I understand that you seem to be looking for a way to change 
types (definitions in general) inside modules you import. This is 
problematic from several aspects, such as other modules depending 
on that module may find that the definitions "change under their 
feet". In D, once a type is declared and its final curly brace is 
closed, you will know that its definition will remain the same 
from anywhere in the program.

D's answer to partial classes is UFCS, however this does not 
allow "adding" fields, only methods.

>     2.4) Original configuration management example would also 
> require the ability to import definitions cyclically. Module A 
> containing ConfigGroupConcrete instantiation imports module B 
> where Config is defined, wich will require B to import A in 
> order to access ConfigGroupConcrete definition.

I don't really understand what you mean here, but D does allow 
cyclic module imports. It is only forbidden when more than one 
module inside any cycle has static constructors, because then it 
is not possible to determine the correct initialization order.

> To conclude, I'll summarize my questions:
> 1). Is there a compiled language that is capable of the 
> abovementiond tricks, without resorting to external templating 
> meta-languages?

I don't know of any. For D, I suggest trying different approaches 
/ paradigms.

> 2). How deep the rabbit hole goes in terms of complexity of 
> preprocessor modifications required? And for DMD in general?

So far, I had not heard of any D project that requires 
preprocessing of D code. I think D's metaprogramming has enough 
solutions to choose from for the vast majority of conceivable 
situations where other languages would call for a preprocessor.

> 4). Is there hope that it's possible to do in, say, a year? I 
> don't mind trying to implement it myself, but I don't want to 
> invest time in thing that is so conceptually out of plane that 
> will simply be too destructive for current compiler environment.

I suggest that you examine how established D projects deal with 
similar situations.