DeRailed DSL (was Re: compile-time regex redux)

[Tom S]

First of all, I totally agree with Sean.

Using the compiler for preprocessing is not a step in the right 
direction. It only gives delusional benefits, while in fact being a 
source of many problems.

Let's face it, the new mixin stuff is a handy tool for generating 
few-line snippets that the current template system cannot handle. Some 
would argue that it's already too much like the C preprocessor, but IMO, 
it's fine.
But going further that way and extending the compiler into a general 
purpose pluggable text processor isn't anything that will make the 
language any more powerful. Not to mention that a standard-compliant D 
compiler was meant to be simple...

When the compiler is used for the processing of a DSL, it simply masks a 
simple step that an external tool would do. It's not much of a problem 
to run an external script or program, that will read the DSL and output 
D code, while perhaps also doing other stuff, connecting with databases 
or making coffee. But when this is moved to the compiler, security 
problems arise, code becomes more cryptic and suddenly, the D code 
generated from the DSL cannot be simply accessed. It's simply produced 
by the 'compiler extension' and given further into compilation. A 
standalone tool will produce a .d module, which can be further verified, 
processed by other tools - such as one that generates reflection data - 
and when something breaks, one can step into the generated source, 
review it and easier spot errors in it. Sean also mentioned that the DSL 
processor will probably need some diagnostic output, and simple 
pragma(msg) and static assert simply won't cut it.

Therefore, I'd like to see a case when a compile-time DSL processing is 
going to be really useful in the real world, as to provoke further 
complication of the compiler and its usage patterns.

The other aspect I observed in the discussions following the new dmd 
release, is that folks are suggesting writing full language parsers, D 
preprocessors and various sort of operations on complex languages, 
notably extended-D parsing... This may sound weird in my lips, but 
that's clearly abuse. What these people really want is a way to extend 
the language's syntax, pretty much as Nemerle does it. And frankly, if D 
is going to be a more powerful language, built-in text preprocessing 
won't cut it. Full fledged macro support and syntax extension mechanics 
are something that we should look at.


--
Tomasz Stachowiak



Sean Kelly wrote:
> What I've done in the past is manage the entire schema, stored 
> procedures and all, in a modeling system like ErWin.  From there I'll 
> dump the lot to a series of scripts which are then applied to the DB. In 
> this case, the DSL would be the intermediate query files, though parsing 
> the complete SQL query syntax (since the files include transactions, 
> etc), sounds sub-optimal.  I suppose a peripheral data format would 
> perhaps be more appropriate for generating code based on the static 
> representation of a DB schema.  UML perhaps?
> 
> My only concern here is that the process seems confusing and unwieldy: 
> manage the schema in one tool, dump the data description in a 
> meta-language to a file, and then have template code in the application 
> parse that file during compilation to generate code.  Each of these 
> translation points creates a potential for failure, and the process and 
> code risks being incomprehensible and unmanageable for new employees.
> 
> Since you've established that the schema for large systems changes only 
> rarely and that the changes are a careful and deliberate process, is it 
> truly the best approach to attempt to automate code changes in this way? 
>  I would think that a well-designed application or interface library 
> could be modified manually in concert with the schema changes to produce 
> the same result, and with a verifiable audit trail to boot.
> 
> Alternately, if the process were truly to be automated, it seems 
> preferable to generate D code directly from the schema management 
> application or via a standalone tool operating on the intermediate data 
> rather than in preprocessor code during compilation.  This approach 
> would give much more informative error messages, and the process could 
> be easily tracked and debugged.
> 
> Please note that I'm not criticizing in-language DSL parsing as a 
> general idea so much as questioning whether this is truly the best 
> example for the usefulness of such a feature.