Is metaprogramming useful?

[Steve Horne]

On Mon, 27 Nov 2006 14:28:15 +0100, "Frank Benoit (keinfarbton)"
<benoit at tionex.removethispart.de> wrote:

>Well, D might be faster, but it shows that the compile time can increase
>very fast.

When doing the metaprogramming, the compiler is basically acting as an
interpreter as opposed to a compiler. It is nothing more advanced or
freaky than that.

A lot of work is already done in interpreted languages already. In
fact, since that term is often applied to languages that actually
compile to virtual machine bytecode (e.g. Python) and since other
languages that compile to virtual machine bytecode are considered
compilers (e.g. Java) there is no clear line between compilers and
interpreters these days. Interpreters often do a bit of pre-compiling
(to intermediate code), and compilers often do a bit of interpreting
(precalculating constant expressions, as well as metaprogramming).

The reason C++ metaprogramming is slow is (1) because it has never
been a big priority for compiler developers, and (2) because C++
template metaprogramming forces programmers to work in indirect and
inefficient ways, substituting specialisation for simple conditionals
and recursion for simple loops.

Support metaprogramming properly and it need be no slower than, for
instance, using a scripting language for code generation - an
annoyingly common thing. In fact, the compiler could simply generate a
compiled code generator, run it, and then pick up the output for the
next compilation stage if performance was that big a deal. This could
give better performance than writing code generation tools in C++,
since the compiler knows its own internal representations and could
create compiled code generators that generate this directly rather
than generating source code.

Getting back to scripting languages, Perl was originally described as
'programmers glue' or 'programmers duct tape' since it was written as
an extension to awk, which was widely used for simple code generation.
The web page generation thing obviously became the real Perl killer
app, but that was a bit later. So clearly code generation is quite a
common necessity, or why would people write tools to make it easier?

People have even used XSLT to generate source code - strange, but
true.

But using a scripting language for code generation is a bad thing,
since it means you have to learn two languages to develop one
application. Why not just have all the tools you need in one language?

>In both cases an external regex or parser generator would make more sense.
>
>Now my questions:
>1.) Is metaprogramming really useful or only kind of hype?
>2.) Are there examples where a metaprogramming solution is the best way
>to solve it?

These are really the same questions, since metaprogramming is useful
if and only if it is the best way to solve certain problems. So...


Consider that the whole point of a library is that as much complexity
as possible should be handled once (in the library itself) rather than
over-and-over again by the users of the library. But at the same time,
that shouldn't add unnecessary run-time overheads. And also, you
shouldn't duplicate code - ideally, each algorithm should be written
once with all the needed flexibility - not in several separately
maintained versions.

Well, even generic data structures can reach a point where you need
'metaprogramming' to handle them properly - where you want lots of
flexibility. Managing bloat can lead to even more complexity, since
different applications require different trade-offs between code size
and speed, so that can be a few more options there.

How many options can there be? Well, maybe you have a particular data
structure that you need, but sometimes you need it in memory and
sometimes you need it in a disk file. Sometimes, you even need it
split between memory and a file (e.g. specialised caching, or handling
transactions with commit and rollback). Sometimes you need maximum
performance and want to do any moves using memcpy, other times you
want to support arbitrary objects and cannot assume that memcpy is
safe so you need to use copy constructors and assignment operators
etc. And maybe there are optional features to your data structures -
e.g. it's a tree data structure that can hold certain kinds of summary
information in nodes to optimise certain search operations, but only
if that summary information is useful for the current application. For
example, you might want efficient subscripted access to an associative
container - easy if each node knows how many items are in its subtree,
impossible otherwise.

My experience is that doing this kind of thing in C++ is easy enough
when dealing with the search and iteration algorithms, but insert and
delete algorithms can get very complex to manage because there are so
many variations. You end up with fragments of each algorithm, each
with multiple variants, selected through specialisation. But thats a
nightmare, so in practice you end up writing several separate
libraries, increasing the bloat, and losing some of the flexibility
that comes from having all the options in one library. Not to mention
the maintenance headache from having several copies of each algorithm.

The D way - each algorithm remaining a single algorithm, but with a
few static-if blocks - is a lot easier. For some things, you can even
prototype with run-time ifs and convert to static-ifs later just by
adding the word 'static' here and there.


Of course this sounds like just making libraries overcomplex, but the
whole point of a library is that as much complexity as possible should
be handled once (in the library itself) rather than over-and-over
again by the users of the library. But at the same time, that
shouldn't add unnecessary run-time overheads.

But then again, the truth is that most programmers simply won't write
libraries to be that flexible because of the complexity. That isn't
how programming should be, though, it's just a limitation of current
programming languages - you can't reasonably do some of the things you
should be able to do easily.


Right now, I don't think I'd use a metaprogramming-based regular
expression or parser library, and certainly not in C++. A lot of
current metaprogramming stuff is experimental, and not really to be
used for serious work. Some of those experiments remind me of the old
obfuscated C contest. But todays experiments are important - without
them, tomorrow will be no better than today.

Or perhaps you'd rather be driving a horse and cart, since after all
the earliest cars were clearly impractical and obviously horses were
fine up until then, which PROOVES there could never be a real need for
cars, eh! ;-)

-- 
Remove 'wants' and 'nospam' from e-mail.