Configurable syntax

[Jason House]

Georg Wrede Wrote:

> 
> --- Disclaimer: this is a bit long. Read it later. ---

Wow, you're right. Sadly, I stopped reading about 80% through (discussion on D4)

At a high level, I agree with your assessment about an easy growth path for newbors to gurus. I should also add that a language must make the code of gurus remain understandable for all but the newest of newbies. (Ever try to read through STL or a boost library?)

I also had an idea for a configurable syntax language that I almost started developing. The concept was dead simple:
1. Provide the ability to call generated code at compile time. D comes close to this, but D has some restrictions on which code can be called.
2. Expose all compiler primitives for potential use in user code: parse trees, types, IL, raw assembly, etc...
3. Provide hooks to replace specific compilation phases (or wedge tasks between phases)

With that, the language was sufficiently capable to morph into any existing language. Everything else, including most language design was "merely" standard library work.

My inspiration was seeing various boost libraries such as spirit and lambda. They seemed like the coolest things ever, but all they did was add new syntax for a common coding need.  I hadn't heard of LINQ but knew such a thing was feasible in my language...




> Andrei Alexandrescu wrote:
> > Yigal Chripun wrote:
> >> Macros in Nemerle need to be compiled since they are regular Nemerle 
> >> code and they need to be loaded by the compiler (added to the command 
> >> line) in order to compile the code the calls the macros.
> >>
> >> essentially these are just plugins for the compiler.
> >>
> >> compared to the elegance of this solution, templates are just a crude 
> >> copy-paste mechanism implemented inside the compiler.
> > 
> > Nemerle's interesting, but it has its own issues. The largest one is 
> > that it will have to beat history: languages with configurable syntax 
> > have failed in droves in the 1970s.
> 
> This actually raises a larger issue.
> 
> IMHO, it's not about configurable syntax, OOP, or any single paradigm.
> 
> What really matters for a language is, the continuum that the 
> newbie--to--guru programmer faces during his ascent through the language.
> 
> A *perfect* language should be reasonably usable by the newbie. While he 
> advances, he will encounter both needs and solutions for his 
> increasingly advanced/complex programming projects. Now, *any* serious 
> discontinuity, or forced change of paradigm, will constitute a hurdle. 
> And each such hurdle only serves as a point where this programmer (maybe 
> momentarily, but still), considers other alternatives. If such a point 
> is conspicuous or steep enough, he'll most probably venture to another 
> language.
> 
> A programming language that has this in mind, will work like Windows: 
> there's always an obvious (who said good?? it's Windows, for crying out 
> loud) way to go, and for you to feel you're advancing on your career.
> 
> > Before I got into D, I was working on Enki. Enki was my own programming 
> > language and of course made D look like a piece of crap. In Enki, you 
> > had only very few primitives related to macro expansion, and you could 
> > construct all language elements - if, while, for, structures, classes, 
> > exceptions, you name it, from those primitive elements.
> 
> Lisp, Forth, anyone???
> 
> > There were two elements that convinced me to quit Enki. One was that I'd 
> > got word of a language called IMP72. IMP72 embedded the very same ideas 
> > Enki had, with two exceptions: (1) it was created in 1972, and (2) 
> > nobody gave a damn ever since. IMP72 (and there were others too around 
> > that time) started with essentially one primitive and then generated 
> > itself with a bootstrap routine, notion that completely wowed me and I 
> > erroneously thought would have the world wowed too.
> 
> Yeah, the academic honey pot. Always looks seductive in TeX formatted 
> papers!
> 
> > The second reason was that I've had many coffees and some beers with 
> > Walter and he convinced me that configurable syntax is an idea that 
> > people just don't like.
> 
> You must forgive him, he's almost as old as I am!!! :O  [1]
> 
> > Thinking a bit more, I realized that humans don't operate well with
> > configurable syntax.
> 
> Actually, it's not only about that. Configurable syntax is of course a 
> major issue. But there's an other one. And that's redundancy.
> 
> In any Real-Life language, there's some amount of redundancy. For 
> example, in many languages, both the predicate and the object are (is 
> this the word) inflected (?), in unison. This gives the ability for the 
> listener to skip (or not hear because of background noise) the ending of 
> either word, and then understand the sentence as a whole.
> 
> In a similar vein, verbs have a different ending with plural and single, 
> even though the "number" is still available from the rest of the 
> sentence, or simply even the context.
> 
> Redundancy lets people receive oral communication without undue fixation 
> on single-phoneme differences in the utterance. One might consider that 
> as a form of error checking, like 9-bit parity memory chips.
> 
> 
> (Something you might want to remember when people ask for compulsory 
> empty parenthesis at the end of functions and methods.) ((See my next 
> paragraph.))
> 
> > To use the hackneyed comparison, no natural language or similar
> > concoction has configurable syntax. Not even musical notation or
> > whatnot. There's one syntax for every human language. I speculated
> > that humans can learn one syntax for a language and then wire their
> > brains to just pattern match semantics using it. Configurable syntax
> > just messes with that approach, and besides makes any program hugely
> > context-dependent and consequently any large program a pile of crap.
> 
> We may have a continuum here. At the one end is the configurable syntax, 
> at the other end the cast-in-stone regular syntax (which often comes 
> with a reduced vocabulary).
> 
> I read (IIRC, Sicentific American, late 1970's) that the average size of 
> vocabulary that a (then) working-class London person in a "lesser" 
> neighbourhood used during a week, was only three hundred words! (This 
> factoid was so shocking to me that I will not forget it till the day I'm 
> on my death-bed. I've got more than 40000 words in each of English, 
> Swedish, and Finnish. [4])
> 
> As an opposite, take this paragraph from an imaginary math book:
> 
>    "Now, substituting a^e*(2pi*sin(alpha[inverse of the second 
> derivative of beta at the index])) for any first term in every 
> polynomial in the quaternion, it becomes obvious that the derivation 
> needs a temporary extra term, which we will call Z(fab).
> 
> Now, after the substitution (which we will use thorugh page 34), we now 
> will study the short-term changes in every dimension we're investigating 
> here."
> 
> Wouldn't you say, this is "configurable syntax". And the poor student 
> doesn't have a choice but to try to follow this crap. (No wonder the 
> working-class guy has no more a chance of ever following this, than the 
> productive, but oh, so challenged member of this newsgroup.)
> 
> 
> All my musings here, make me write two things:
> 
> 1: A language has to *provide* an obvious (and followable) path from the 
> ridiculously simple, to the ultimately esoteric, with no (or as few as 
> possible) discontinuities in mind set. (But see [2].)
> 
> 2: Any _categorical_ statement that contains an implicit assessment of 
> the {audience's|programmer's|critic's} abilities, and their relevance to 
> the level that a particular construct in a language demands, is 
> inherently obsolete. An example is "configurable syntax is an idea that 
> people just don't like".
> 
> > That being said, I have no idea whether or not Nemerle will be 
> > successful. I just speculate it has an uphill battle to win.
> 
> We better make sure D doesn't climb the same hill.
> 
> And damn sure that the potholes in the D uphill aren't larger than the 
> disciples' stride.
> 
> 
> ---------
> 
> [1] Configurable syntax belongs to the same group as recursion. You can 
> program for years without ever needing it (or understanding that you did 
> need it, actually). And then, you see it in a textbook, and all they 
> used it for was Fibonacci. Gee, great, you think, and then you write 
> your own Fibonacci, and then just forget about the whole thing. They 
> simply don't (or didn't, at the time I read textbooks on programming) 
> ever show its relevance in real life everyday programming. (Yes, maybe 
> at that time, hard disks with directory hierarchies were not common. (I 
> used CP/M, couldn't afford UNIX. Neither did they.) Otherwise they'd all 
> have had an example of summing the sizes of files in a directory tree. 
> Which would at once have made the reader feel recursion actually has 
> some relevance to their life.)
> 
> Configurable syntax isn't exactly what Walter makes his bread and butter 
> on. Neither do I. So neither of us has any reason (today) to use it, or 
> even imagine the benefits of it.
> 
> 
> Having said that, I remember the time (years ago) when I suggested in 
> this newsgroup that we should have a macro language above the regular 
> programming syntax. I suggested it would be somewhat like Lisp, and it 
> would then let us do some most amazing things, totally inconceivable to 
> us at the time. I almost became the laughing stock of the crowd. (And 
> those things were (mostly) much less amazing than what D templates do 
> now. Except for a little configurable syntax stuff. :-) )
> 
> Similarily, I see configurable syntax as a way (somewhat orthogonal to 
> templates, but not fully) of making it possible to *easily* write some 
> kinds of programs that we really can't imagine right now.
> 
> 
> As an example of how hard it is to get this stuff across, I'll admit to 
> the following: in the old days, I used to imagine that Lisp is the 
> all-powrful super language for _anything_ (except for bit twiddling and 
> systems work), and that if you couldn't do something in Lisp, it was 
> only because of your own limitations as a person.
> 
> Then, one day, I read that they're implementing macros! I dropped from 
> my chair! If there ever was a language I didn't expect to need macros, 
> Lisp definitely was it. Then, during the next years I read articles that 
> made me believe (I wouldnt be preposterous enough to state I 
> /understood/) that they actually were needed, and that they advanced the 
> applicability of the language significantly.
> 
> 
> So, configurable syntax for D, I assume *will* be a reality. But I hope 
> it won't be before 2.0 is official. :-( !!!!
> 
> And while I'm at it, I think that in D4 (not D3) we will have a total 
> overhaul of the current template syntax. (!!!) Yes, it beats the 
> G-string off C++, no question. But that is not enough. And, when you 
> really look at it, the "ease" of D templates is an illusion. The "D like 
> syntax", the "invisibility" of the template system, is only to appease 
> newcomers, to piss off C++ folks, and to delude ourselves.
> 
> Time will show that it will need to be more overtly Lisp like, and that 
> actually a (sort of) difference in syntax to regular D is well motivated.
> 
> 
> [2] This is in contrast to the "uphill potholes" I was talking before. 
> Why it's okay (IMHO) to have a slightly different syntax, is, the mental 
> leap from regular programming to templates is huge anyway. And it can't 
> be made to go away either[3]. (Rumtime versus compile time is already 
> hard enough for many. Then we have "is my code producing binary code, or 
> is it producing some other source code? Then what happens? Can that in 
> turn produce yet more source code?? Am I forced to only use this 
> Recursion Stuff all over the place?", etc.)
> 
> So, then it would only be natural to have a slightly different syntax. 
> That would also give us possiblilities beyond the current ones, to 
> (first partly, and then more completely) implement a system of 
> configurable syntax to D.
> 
> 
> In the same vein that D still implements an ASM language, there should 
> be a (ehh...) "Higher Level" language at the other end. Precisely like D 
> has never deluded itself of being efficient enough at the Very Low Level 
> to really make ASM obsolete, it should not delude itself with notions of 
> not needing a fundamental VHL (Very High Level) language at the other end.
> 
> The current Template Language is excellent, beats anybody in sight, but 
> that's not the final answer. And before long C++@#$@#$@ will have 
> implemented it, too. Or at least the more marketable details of it. But 
> we have to maintain our lead. And that's done by advancing the bleeding 
> edge so it stays just beyond their reach.
> 
> 
> [3] One can try to hide a difference by making something "look the 
> same". But that only serves to disorient people. Do you remember that 
> there were *two* mazes in DD. Way past the "xyzzy" graffiti, you ended 
> up in a maze. I spent a whole week around there, before a girlfriend 
> pointed out that "the corridors are alike" but yesterday "the corridors 
> are all different". That's when I understood there were two mazes. No 
> wonder I'd had a feeling of being totally clueless there.
> 
> Microsoft has made a career of making stuff look alike when they're not, 
> and non-alike when they are. It's poison to rational, intelligent users. 
> But the idiots who just learn by rote memory what to do when, couldn't 
> care less. Or understand the point of our grief.
> 
> 
> [4] How would you measure how many words you have in a language? The 
> easiest way (back when there were books), was to have somebody else look 
> up a random word in a random page of a dictionary, and ask you to 
> explain it. Do that like 80 times, and then calculate the percentage you 
> got right. Compare that to how many words the dictionary brags of 
> having, usually right on the cover. (You do need a proper dictionary, 
> with more than 50000 words, of course.)
>