Configurable syntax

[Georg Wrede]

--- Disclaimer: this is a bit long. Read it later. ---

Andrei Alexandrescu wrote:
> Yigal Chripun wrote:
>> Macros in Nemerle need to be compiled since they are regular Nemerle 
>> code and they need to be loaded by the compiler (added to the command 
>> line) in order to compile the code the calls the macros.
>>
>> essentially these are just plugins for the compiler.
>>
>> compared to the elegance of this solution, templates are just a crude 
>> copy-paste mechanism implemented inside the compiler.
> 
> Nemerle's interesting, but it has its own issues. The largest one is 
> that it will have to beat history: languages with configurable syntax 
> have failed in droves in the 1970s.

This actually raises a larger issue.

IMHO, it's not about configurable syntax, OOP, or any single paradigm.

What really matters for a language is, the continuum that the 
newbie--to--guru programmer faces during his ascent through the language.

A *perfect* language should be reasonably usable by the newbie. While he 
advances, he will encounter both needs and solutions for his 
increasingly advanced/complex programming projects. Now, *any* serious 
discontinuity, or forced change of paradigm, will constitute a hurdle. 
And each such hurdle only serves as a point where this programmer (maybe 
momentarily, but still), considers other alternatives. If such a point 
is conspicuous or steep enough, he'll most probably venture to another 
language.

A programming language that has this in mind, will work like Windows: 
there's always an obvious (who said good?? it's Windows, for crying out 
loud) way to go, and for you to feel you're advancing on your career.

> Before I got into D, I was working on Enki. Enki was my own programming 
> language and of course made D look like a piece of crap. In Enki, you 
> had only very few primitives related to macro expansion, and you could 
> construct all language elements - if, while, for, structures, classes, 
> exceptions, you name it, from those primitive elements.

Lisp, Forth, anyone???

> There were two elements that convinced me to quit Enki. One was that I'd 
> got word of a language called IMP72. IMP72 embedded the very same ideas 
> Enki had, with two exceptions: (1) it was created in 1972, and (2) 
> nobody gave a damn ever since. IMP72 (and there were others too around 
> that time) started with essentially one primitive and then generated 
> itself with a bootstrap routine, notion that completely wowed me and I 
> erroneously thought would have the world wowed too.

Yeah, the academic honey pot. Always looks seductive in TeX formatted 
papers!

> The second reason was that I've had many coffees and some beers with 
> Walter and he convinced me that configurable syntax is an idea that 
> people just don't like.

You must forgive him, he's almost as old as I am!!! :O  [1]

> Thinking a bit more, I realized that humans don't operate well with
> configurable syntax.

Actually, it's not only about that. Configurable syntax is of course a 
major issue. But there's an other one. And that's redundancy.

In any Real-Life language, there's some amount of redundancy. For 
example, in many languages, both the predicate and the object are (is 
this the word) inflected (?), in unison. This gives the ability for the 
listener to skip (or not hear because of background noise) the ending of 
either word, and then understand the sentence as a whole.

In a similar vein, verbs have a different ending with plural and single, 
even though the "number" is still available from the rest of the 
sentence, or simply even the context.

Redundancy lets people receive oral communication without undue fixation 
on single-phoneme differences in the utterance. One might consider that 
as a form of error checking, like 9-bit parity memory chips.


(Something you might want to remember when people ask for compulsory 
empty parenthesis at the end of functions and methods.) ((See my next 
paragraph.))

> To use the hackneyed comparison, no natural language or similar
> concoction has configurable syntax. Not even musical notation or
> whatnot. There's one syntax for every human language. I speculated
> that humans can learn one syntax for a language and then wire their
> brains to just pattern match semantics using it. Configurable syntax
> just messes with that approach, and besides makes any program hugely
> context-dependent and consequently any large program a pile of crap.

We may have a continuum here. At the one end is the configurable syntax, 
at the other end the cast-in-stone regular syntax (which often comes 
with a reduced vocabulary).

I read (IIRC, Sicentific American, late 1970's) that the average size of 
vocabulary that a (then) working-class London person in a "lesser" 
neighbourhood used during a week, was only three hundred words! (This 
factoid was so shocking to me that I will not forget it till the day I'm 
on my death-bed. I've got more than 40000 words in each of English, 
Swedish, and Finnish. [4])

As an opposite, take this paragraph from an imaginary math book:

   "Now, substituting a^e*(2pi*sin(alpha[inverse of the second 
derivative of beta at the index])) for any first term in every 
polynomial in the quaternion, it becomes obvious that the derivation 
needs a temporary extra term, which we will call Z(fab).

Now, after the substitution (which we will use thorugh page 34), we now 
will study the short-term changes in every dimension we're investigating 
here."

Wouldn't you say, this is "configurable syntax". And the poor student 
doesn't have a choice but to try to follow this crap. (No wonder the 
working-class guy has no more a chance of ever following this, than the 
productive, but oh, so challenged member of this newsgroup.)


All my musings here, make me write two things:

1: A language has to *provide* an obvious (and followable) path from the 
ridiculously simple, to the ultimately esoteric, with no (or as few as 
possible) discontinuities in mind set. (But see [2].)

2: Any _categorical_ statement that contains an implicit assessment of 
the {audience's|programmer's|critic's} abilities, and their relevance to 
the level that a particular construct in a language demands, is 
inherently obsolete. An example is "configurable syntax is an idea that 
people just don't like".

> That being said, I have no idea whether or not Nemerle will be 
> successful. I just speculate it has an uphill battle to win.

We better make sure D doesn't climb the same hill.

And damn sure that the potholes in the D uphill aren't larger than the 
disciples' stride.


---------

[1] Configurable syntax belongs to the same group as recursion. You can 
program for years without ever needing it (or understanding that you did 
need it, actually). And then, you see it in a textbook, and all they 
used it for was Fibonacci. Gee, great, you think, and then you write 
your own Fibonacci, and then just forget about the whole thing. They 
simply don't (or didn't, at the time I read textbooks on programming) 
ever show its relevance in real life everyday programming. (Yes, maybe 
at that time, hard disks with directory hierarchies were not common. (I 
used CP/M, couldn't afford UNIX. Neither did they.) Otherwise they'd all 
have had an example of summing the sizes of files in a directory tree. 
Which would at once have made the reader feel recursion actually has 
some relevance to their life.)

Configurable syntax isn't exactly what Walter makes his bread and butter 
on. Neither do I. So neither of us has any reason (today) to use it, or 
even imagine the benefits of it.


Having said that, I remember the time (years ago) when I suggested in 
this newsgroup that we should have a macro language above the regular 
programming syntax. I suggested it would be somewhat like Lisp, and it 
would then let us do some most amazing things, totally inconceivable to 
us at the time. I almost became the laughing stock of the crowd. (And 
those things were (mostly) much less amazing than what D templates do 
now. Except for a little configurable syntax stuff. :-) )

Similarily, I see configurable syntax as a way (somewhat orthogonal to 
templates, but not fully) of making it possible to *easily* write some 
kinds of programs that we really can't imagine right now.


As an example of how hard it is to get this stuff across, I'll admit to 
the following: in the old days, I used to imagine that Lisp is the 
all-powrful super language for _anything_ (except for bit twiddling and 
systems work), and that if you couldn't do something in Lisp, it was 
only because of your own limitations as a person.

Then, one day, I read that they're implementing macros! I dropped from 
my chair! If there ever was a language I didn't expect to need macros, 
Lisp definitely was it. Then, during the next years I read articles that 
made me believe (I wouldnt be preposterous enough to state I 
/understood/) that they actually were needed, and that they advanced the 
applicability of the language significantly.


So, configurable syntax for D, I assume *will* be a reality. But I hope 
it won't be before 2.0 is official. :-( !!!!

And while I'm at it, I think that in D4 (not D3) we will have a total 
overhaul of the current template syntax. (!!!) Yes, it beats the 
G-string off C++, no question. But that is not enough. And, when you 
really look at it, the "ease" of D templates is an illusion. The "D like 
syntax", the "invisibility" of the template system, is only to appease 
newcomers, to piss off C++ folks, and to delude ourselves.

Time will show that it will need to be more overtly Lisp like, and that 
actually a (sort of) difference in syntax to regular D is well motivated.


[2] This is in contrast to the "uphill potholes" I was talking before. 
Why it's okay (IMHO) to have a slightly different syntax, is, the mental 
leap from regular programming to templates is huge anyway. And it can't 
be made to go away either[3]. (Rumtime versus compile time is already 
hard enough for many. Then we have "is my code producing binary code, or 
is it producing some other source code? Then what happens? Can that in 
turn produce yet more source code?? Am I forced to only use this 
Recursion Stuff all over the place?", etc.)

So, then it would only be natural to have a slightly different syntax. 
That would also give us possiblilities beyond the current ones, to 
(first partly, and then more completely) implement a system of 
configurable syntax to D.


In the same vein that D still implements an ASM language, there should 
be a (ehh...) "Higher Level" language at the other end. Precisely like D 
has never deluded itself of being efficient enough at the Very Low Level 
to really make ASM obsolete, it should not delude itself with notions of 
not needing a fundamental VHL (Very High Level) language at the other end.

The current Template Language is excellent, beats anybody in sight, but 
that's not the final answer. And before long C++@#$@#$@ will have 
implemented it, too. Or at least the more marketable details of it. But 
we have to maintain our lead. And that's done by advancing the bleeding 
edge so it stays just beyond their reach.


[3] One can try to hide a difference by making something "look the 
same". But that only serves to disorient people. Do you remember that 
there were *two* mazes in DD. Way past the "xyzzy" graffiti, you ended 
up in a maze. I spent a whole week around there, before a girlfriend 
pointed out that "the corridors are alike" but yesterday "the corridors 
are all different". That's when I understood there were two mazes. No 
wonder I'd had a feeling of being totally clueless there.

Microsoft has made a career of making stuff look alike when they're not, 
and non-alike when they are. It's poison to rational, intelligent users. 
But the idiots who just learn by rote memory what to do when, couldn't 
care less. Or understand the point of our grief.


[4] How would you measure how many words you have in a language? The 
easiest way (back when there were books), was to have somebody else look 
up a random word in a random page of a dictionary, and ask you to 
explain it. Do that like 80 times, and then calculate the percentage you 
got right. Compare that to how many words the dictionary brags of 
having, usually right on the cover. (You do need a proper dictionary, 
with more than 50000 words, of course.)