DIP 50 - AST macros

[Walter Bright]

On 11/10/2013 1:20 PM, Jacob Carlborg wrote:
> I've been thinking quite long of how AST macros could look like in D. I've been
> posting my vision of AST macros here in the newsgroup a couple of times already.
> I've now been asked to create a DIP out of it, so here it is:
>
> http://wiki.dlang.org/DIP50


I confess I have some serious reservations about AST macros in general:

1. I've seen very heavy use of such macros in macro assemblers. What happens is 
people use it to invent their own (very baroque) language on top of the existing 
assembler one. Anyone trying to read the code has to learn this new unique 
language, and given the limitations of the macro capability, it often comes with 
unfixable bizarre corner cases and limitations.

This got so bad that in some cases I know of, the poor sap who is given the job 
of making improvements resorted to running the assembler to generate an object 
file, then disassembling the object file back into source code! Something went 
very wrong for this to be necessary. I know in my own assembler work I have 
abandoned all use of macros.

(Note that macros in typical assemblers are far more powerful than C's macro 
language, which stands out for being pathetically underpowered.)

2. Macros are something that should be used very sparingly. However, this is not 
what happens. I used to be perplexed that even top shelf C/C++ programmers tend 
to write what I not so humbly consider to be pretty abusive use of macros. Now 
it just saddens me.

3. Lisp is a language that encourages users to write macros to pretty much 
invent a custom language for the task at hand. Heavy use of such makes the code 
unrecognizable as being Lisp code. I believe a language ought to have some 
"anchors" that the person reading the code can reliably recognize. (GO has taken 
this idea pretty far.)

4. AST macros, pretty much by definition, manipulate the AST. However, if you 
examine the semantic routines in DMD, a *lot* more is required to do more 
significant things. The symbol table must be consulted, etc. I just don't see 
how one could hope to implement something like function overloading with AST 
macros. Or template type deduction. Or closure semantics, attribute inference, 
etc. Sure, some forms of foreach can be rewritten into for statements by 
manipulating the AST, but not the more interesting forms, such as the ones that 
deal with opApply().

There are some statements in the DIP about the Context parameter and it'll 
provide semantic information, but I don't see how this can work without making 
it an ever-expanding collection of arbitrary methods.

5. We've said "no" in the past to things like user-defined tokens, which are 
closely related.

6. Note that D has a limited form of AST macros with mixin templates.


To sum up, any AST macro proposal has some tough barriers to get over. Within 
its necessarily severe semantic limitations, it has to deliver a pretty 
compelling set of solutions, while avoiding the morass of incomprehensibility 
that far too many macro systems become in practice.

For example, the Linq example in the DIP is not compelling, as aesthetically 
nicer code can be written using D's ranges and algorithms:

     auto data = arr.filter!(x => x > 5).array;

I see no compelling advantage in trying to make D code look like C#; to be blunt 
it's like the old:

     #define BEGIN {
     #define END }

macros used in old C code to make it look like Pascal.