Whole source-tree statefull preprocessing, notion of a whole program

[Boris-Barboris via Digitalmars-d]

On Saturday, 8 April 2017 at 13:09:59 UTC, Vladimir Panteleev 
wrote:
> On Saturday, 8 April 2017 at 10:11:11 UTC, Boris-Barboris wrote:

>> 2). After preprocessing I wish to have fully-typed, safe and 
>> fast Config class, that contains all the groups I defined for 
>> it in it's body, and not as references. I don't want pointer 
>> lookup during runtime to get some field.
>
> Looks like your current implementation does not go in that 
> direction, seeing as it uses properties for field access.

Am i mistaken in assumption that such simple getter property will 
be optimized to direct field access? Anyways, that's minor detail.

> For such tasks, I would suggest to split the representations 
> (native data, DOM tree, JSON strings etc.) from the 
> transformations (serialization and parsing/emitting JSON). E.g. 
> your example could be represented as:
>
> struct Config
> {
>     struct TestGroup
>     {
>         string some_string_option;
>         double some_double_option;
>     }
>     TestGroup testGroup;
> }
>
> Then, separate code for serializing/deserializing this to/from 
> a DOM or directly to/from JSON.
>
> Individual components' configuration can be delegated to their 
> components; their modules could contain public struct 
> definitions that you can add to the global Config struct, which 
> describes the configuration of the entire application. I've 
> used this pattern successfully in some projects, incl. Digger: 
> https://github.com/CyberShadow/Digger/blob/master/config.d#L31-L36

Ok, that's nice, but it still requires manual inclusion of such 
field into global config struct. Some "compile-time callback" 
system still would scale better in my opinion.

>>     2.2) Sweet dream of 2.1 is met with absence of tools to 
>> create and manipulate state during preprocessing. For example:
>
> I understand that you seem to be looking for a way to change 
> types (definitions in general) inside modules you import. This 
> is problematic from several aspects, such as other modules 
> depending on that module may find that the definitions "change 
> under their feet".

As expected since class that allows itself to be modified in 
compile-time, always does so explicitly via mixin. Most of the 
times such manipulation is used to extend functionality (add 
field, plugin, method) without removing or modifying existing 
ones. And if the names conflict, we get nice compile-time error 
anyways.

> In D, once a type is declared and its final curly brace is 
> closed, you will know that its definition will remain the same 
> from anywhere in the program.

That's kinda my point - definition needs to stay the same because 
it's built by compiler as many times as there are transtaltion 
units, because evil old C grandpa.

> D's answer to partial classes is UFCS, however this does not 
> allow "adding" fields, only methods.

Adding fields, or, generally, objects \ collections of objects, 
is the main use case. Adding methods in my experience is rare 
scenario.

>>     2.4) Original configuration management example would also 
>> require the ability to import definitions cyclically. Module A 
>> containing ConfigGroupConcrete instantiation imports module B 
>> where Config is defined, wich will require B to import A in 
>> order to access ConfigGroupConcrete definition.
>
> I don't really understand what you mean here, but D does allow 
> cyclic module imports. It is only forbidden when more than one 
> module inside any cycle has static constructors, because then 
> it is not possible to determine the correct initialization 
> order.

Exactly what I wanted to know, thank you.

>> To conclude, I'll summarize my questions:
>> 1). Is there a compiled language that is capable of the 
>> abovementiond tricks, without resorting to external templating 
>> meta-languages?
>
> I don't know of any. For D, I suggest trying different 
> approaches / paradigms.
>
>> 2). How deep the rabbit hole goes in terms of complexity of 
>> preprocessor modifications required? And for DMD in general?
>
> So far, I had not heard of any D project that requires 
> preprocessing of D code. I think D's metaprogramming has enough 
> solutions to choose from for the vast majority of conceivable 
> situations where other languages would call for a preprocessor.
>
>> 4). Is there hope that it's possible to do in, say, a year? I 
>> don't mind trying to implement it myself, but I don't want to 
>> invest time in thing that is so conceptually out of plane that 
>> will simply be too destructive for current compiler 
>> environment.
>
> I suggest that you examine how established D projects deal with 
> similar situations.

Thank you.