Metaprogramming in D : Some Real-world Examples

[Bill Baxter]

Hi Nick,

Thanks for the response.  More below.

On Mon, Nov 9, 2009 at 6:46 PM, Nick Sabalausky <a at a.a> wrote:
> "Bill Baxter" <wbaxter at gmail.com> wrote in message
> news:mailman.290.1257812868.20261.digitalmars-d-announce at puremagic.com...
>> On Mon, Nov 9, 2009 at 4:09 PM, Walter Bright
>> <newshound1 at digitalmars.com> wrote:
> I have a few things, mostly part of my SemiTwist D Tools project
> (http://www.dsource.org/projects/semitwist/):
>
> === Trace / Trace Value ===
>
> Two of my personal favorites (particularly traceVal). Great for debugging.
>
> Implementation and Documentation (search for "trace" and "traceVal",
> implementations are less than 10 lines each):
> http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/util/mixins.d
>
> Quick Example:
> -----------------------------
> mixin(trace!());
> mixin(trace!("--EASY TO VISUALLY GREP--"));
>
> int myVar=100;
> mixin(traceVal!("myVar  "));
> mixin(traceVal!("myVar-1"));
> mixin(traceVal!("min(4,7)", "max(4,7)"));
> -----------------------------
>
> Output:
> -----------------------------
> C:\path\file.d(1): trace
> --EASY TO VISUALLY GREP--: C:\path\file.d(3): trace
> myVar  : 100
> myVar-1: 99
> min(4,7): 4
> max(4,7): 7
> -----------------------------

This does, look handy, but the C++ equivalent based on preprocessor
macros arguably looks better and has a simpler implementation:
    #define traceVal(x) cout << #x " : " <<  x; << endl;
and to use just:
    traceVal(myVar);

(assuming for a moment that c++ had a writefln)

However, the unescape() function used in traceVal looks like it might
be a useful CTFE example.  Some kind of compile-time string
escaping/unescaping could definitely be a good example.

> There's also a traceMixin that can be useful for debugging mixins (it'll
> replace a normal string mixin and echo at compile time via "pragma(msg, )"
> the string being mixed in).
>
> === Init Member ===
>
> Great for DRY in constructors with many initialization parameters.
>
> Implementation and Documentation (at the top of the file):
> http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/util/mixins.d
>
> Quick Example:
> -----------------------------
> mixin(initMember!(someVar));
> mixin(initMember!(a, b, c));
> -----------------------------
>
> Turns Into:
> -----------------------------
> this.someVar = someVar;
> this.a = a;
> this.b = b;
> this.c = c;
> -----------------------------
>
> Some variations are also available, such as initMemberFrom for copy
> constructors and initFrom copying class members to local vars.

I think I'd rather focus on examples that let you do something that
would be difficult to do otherwise, rather than something that saves a
bit of typing.

> === Getters ===
>
> DRY mixins for publicly read-only properties, and a "lazy" version for
> lazily computed & cached read-only properties. A poor replacement for a real
> DRY property syntax, but the next best thing.
>
> Implementation and Documentation (search for "getter" and "getterLazy"):
> http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/util/mixins.d
>
> Quick Example:
> -----------------------------
> // Third param optional
> mixin(getter!(float, "someFloat", 2.5));
>
> mixin(getterLazy!(int, "myVar"));
> private int _myVar_gen()
> {
>    // Ordinarily, this function would be much more complex
>    // Also, any member func can set "_myVar_cached = false;"
>    // to force this to be re-computed on the next request.
>    return 7;
> }
> -----------------------------
>
>
> Turns Into:
> -----------------------------
> private float _someFloat = 2.5;
> private float someFloat(float _NEW_VAL_)
> {
>    _someFloat = _NEW_VAL_;
>    return _someFloat;
> }
> public float someFloat()
> {
>    return _someFloat;
> }
>
> private int _myVar;
> private bool _myVar_cached = false;
> public int myVar() {
>    if(!_myVar_cached) {
>        _myVar_cached = true;
>        _myVar = _myVar_gen();
>    }
>    return _myVar;
> }
> private int _myVar_gen()
> {
>    // Ordinarily, this function would be much more complex
>    // Also, any member func can set "_myVar_cached = false;"
>    // to force this to be re-computed on the next request.
>    return 7;
> }
> -----------------------------
>
> Variations are also available to use "protected" (or anything else) instead
> of private.

This is more interesting than the last, but still falls pretty much in
the "saves typing" category.  Also I think it's arguable that these
sacrifice some readablility to gain their terseness.  For this sort of
thing I'd rather have an IDE save me the typing, but actually create
the code, instead of resorting to semi-cryptic naming conventions.

> === Defer Assert/Ensure ===
>
> Uses metaprogramming to create an alternative to assert() that provides much
> of the usefulness of JUnit-style libs, but without the bulk of wrapping
> things like '==' and '||' in classes/structs or forcing unnatural syntax
> like '(a.equals(b)).or(c.notEquals(d))'. Also reports unexpected exceptions
> and allows requiring a particular exception to be thrown. Implemented in
> less than 200 lines, including blank lines and comments.
>
> Implementation:
> http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/util/deferAssert.d
>
> Simple Test App:
> http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/apps/tests/deferAssertTest/main.d
>
> Quick Example:
> -----------------------------
> int foo = 2;
> mixin(deferAssert!(`foo == 3 || foo > 5`, "foo is bad"));
> mixin(deferEnsure!(`foo`, `_ == 3 || _ > 5`, "ensure foo failed"));
> flushAsserts();
> -----------------------------
>
> Output:
> -----------------------------
> src\semitwist\apps\tests\deferAssertTest\main.d(37): Assert Failed (foo == 3
> || foo > 5): foo is bad
> src\semitwist\apps\tests\deferAssertTest\main.d(42): Ensure Failed: ensure
> foo failed
> Expression 'foo':
> Expected: _ == 3 || _ > 5
> Actual: 2
> tango.core.Exception.AssertException at src\semitwist\util\deferAssert.d(170):
> 2 Assert Failures
> -----------------------------


This is another one that C++ people would just throw macros at and
wonder what the big deal is.


> === Compile-time checking on types with non-identifier strings ===
>
> Another of my favorites. This is from an upcoming release of my Goldie (
> http://www.dsource.org/projects/goldie ) parser library.
>
> I have a class to represent a token, which could have any name like "+" or
> "Statement", etc., all depending on the user-defined grammar. Currently,
> this token name representation is dynamic/run-time, which is very flexible,
> but there's no compile-time checking of these names, which often a bad
> tradeoff because many programs that need to parse something already know (in
> fact, *must* already know) the token names, so the flexibility doesn't help
> them at all, but compile-time checking would.
>
> Unfortunately, the traditional approach to adding static checking would
> require every token name to be a valid identifier name, so for instance, "+"
> would have to be changed to "Plus", etc. And this can be problematic. It's
> extra work. It's very difficult to automate while getting good well-named
> results. It's possible (in a company setting, for instance) that the
> programmer might not be allowed or able to change the grammar. And of
> course, there's the possibility for name collisions (ie, what if "Plus" was
> already taken by another token?). Also, the new static class definition
> would probably need to be duplicated for each token. But all those problems
> can be solved in D quite easily with just a very small amount of some very
> simple metaprogramming.
>
> The actual implementation Goldie uses does get fairly involved, but the
> basic concept is dead simple. All it takes is one templated type paramater,
> one static if and one static assert (and the static if could easily be
> eliminated):
>
> -------------------------------------------------
> // ----------- Dynamic Token -----------
>
> // A Dynamic token. Flexible, but no compile-time checking
> class DynamicToken
> {
>    protected char[] name;
>    this(char[] name)
>    {
>        if(name != "+" && name != "Statement")
>            throw new Exception("Invalid token: " ~ name);
>
>        this.name = name;
>    }
>    // More Tokeny stuff here
> }
> unittest
> {
>    auto t1 = new DynamicToken("+");
>    auto t2 = new DynamicToken("Statement");
>    auto t3 = new DynamicToken("Stment"); // Runtime error
> }
>
> // ----------- Old-Style Static Token -----------
>
> class OldStyle_StaticToken_Statement : DynamicToken
> {
>    this() { super("Statement"); }
> }
> // Note that "+" must be changed to "Plus",
> // Also, the whole class definition has to be repeated
> // (Although it's pretty small in this case.)
> class OldStyle_StaticToken_Plus : DynamicToken
> {
>    this() { super("+"); }
> }
> unittest
> {
>    // Supposed to be +, but had to be changed to Plus
>    // by either the grammar author or by special complex
>    // logic in the StaticToken-generation tool.
>    // And if there was already another token actually named "Plus"...
>    // Well, even more complexity arises.
>
>    auto t1 = new OldStyle_StaticToken_Plus;
>    auto t2 = new OldStyle_StaticToken_Statement;
>    //auto t3 = new OldStyle_StaticToken_Stment; // Compile-time error
> }
>
> // ----------- New-Style Static Token -----------
>
> class NewStyle_StaticToken(char[] staticName) : DynamicToken
> {
>    static if(staticName != "+" && staticName != "Statement")
>        static assert(false, "Invalid token: " ~ staticName);
>
>    this() { super(staticName); }
> }
> unittest
> {
>    auto t1 = new NewStyle_StaticToken!("+");
>    auto t2 = new NewStyle_StaticToken!("Statement");
>    //auto t3 = new NewStyle_StaticToken!("Stment"); // Compile-time error
> }
> -------------------------------------------------

That is kinda neat, I suppose you could even extend that to make the
list of valid tokens be a template parameter (either variadic, or
separated by some delimiter in one big string that then gets broken up
at runtime).  I guess you could pitch this as a baby step towards the
full-fledged compile-time parser.  Step 0 recognize valid tokens.

Thanks for writing all these up.

--bb