Super-dee-duper D features

[Kevin Bealer]

Great post;  all of this is interesting, and I think I agree with all of 
the individual points.

Walter Bright wrote:
> kris wrote:
...
> 
> Some comments:
...
> 4) The more experience I have, the more it seems that the language that 
> got a lot right is ... Lisp. But Lisp did one thing terribly, terribly 
> wrong - the syntax. The Lisp experts who can get past that seem to be 
> amazingly productive with Lisp. The rest of us will remain envious of 
> what Lisp can do, but will never use it.
> 
> 5) Lisp gets things right, according to what I've read from heavy Lisp 
> users, by being a language that can be modified on the fly to suit the 
> task at hand, in other words, by having a customizable language one can 
> achieve dramatic productivity gains.

Yes - I've often wished for Ocaml or LISP with C syntax, but LISP has 
another real issue to me, which is that 'mutable' algorithms are hard.

> 6) If I think about it a certain way, it looks like what C++ Boost is 
> doing is a desperate attempt to add Lisp-like features. By desperate I 
> mean that C++'s core feature set is just inadequate to the task. For 
> example, look at all the effort Boost has gone through to do a barely 
> functioning implementation of tuples. Put tuples into the language 
> properly, and all that nasty stuff just falls away like a roofer peeling 
> off the old shingles.
...
> 9) But I see what C++ templates can do. So to me, the problem is to 
> design templates in such a way that they are as simple to write as 
> ordinary functions. *Then*, what templates can do can be accessible and 
> maintainable. It's like cars - they used to be very difficult to drive, 
> but now anyone can hop in, turn the key, and go.
...
> 11) Today's way-out feature is tomorrow's pedestrian code. I'm old 
> enough to remember when "structured code", i.e. while, for, switch 
> instead of goto, was the way-out feature (70's). Then, OOP was all the 
> rage (80's), now that's a major yawner. STL was then the way-out fad 
> (90's), now that's pedestrian too. Now it's metaprogramming (00's), and 
> I bet by 2015 that'll be ho-hum too, and it's D that's going to do it.
...

Code in C++ wants to be iterative/imperative, and code in LISP wants to 
be recursive and functional.  But the 'macro system' in both cases is 
written as a functional / recursive system.  In LISP, this was by design 
and works with the language, in C++ its because recursion happened to 
fit through a keyhole in the template grammer.

LISP makes it hard / awkward to work with large mutable structures. 
Everything is functional so a functional macro system can do everything.

C++ templates were great (by today's standards, adequate) for container 
classes, but that task could almost be done just with text substitution.

So...

In my view, to make metaprogramming work in a way that non-LISP people 
understand in something like C++ or D, we need to see programs as a 
series of iterations:

1. A program with all the static code, but main() is replaced by calls
    to functions in each module that manufacture code (code factories).

2. Same as 1, with the newly manufactured code added, along with new
    calls to code factories.

3. Same as 2, etc.

N. A program with all static (non-template) code.

Each step is run to produce the next step -- in most cases, this 
consists of running one code generator after another -- the individual 
steps are not actually explicit, but it is important for one code 
generator to be able to use the results of another when it needs to.

All steps except N are optional, and usually you would have just 1 and 
N.  It might be a good idea at first to require that.

The difference between this world order and the current (C++) way, is 
that in C++ the factories are defined as recursive templates that expand 
like fractals.

Rationale:

LISP macros have access to essentially all of LISP.  If they didn't have 
access to (for instance) mapcar, the LISP macro system would be 
measurably harmed.

D programs need the same consideration --- if a 'regular D' programmer 
had to do without std.string, they would feel the absence every time he 
needed find().  Similarly, every time I try to write a meta-program in D 
today, I feel the same absence of std.string.

I think to really write clean code for the 'code factory' step, I need 
to be able to write nested loops over the user input, accumulate code in 
strings with "~=", and so on without writing it recursively.

My first thought on "how" is to build an interpreter in the compiler 
that can run the parse tree of a D function.  Which brings us fairly 
close to the LISP domain, the primary difference being that this 
interpreter would not necessarily be included in the compiled program 
(i.e. its not 'eval()', yet.)

(Without having written a real working compiler, I don't know how 
practical this is; I think I've heard of parse-tree based interpreters 
in relation to perl.  A reasonable restriction might be "no C calls".)

Kevin