Unit tests in D

[Tomek Sowiński]

Dnia 05-05-2010 o 03:24:50 bearophile <bearophileHUGS at lycos.com>  
napisał(a):

[snip]
> 2) Names for unittests. Giving names to things in the universe is a  
> first essential step if you want to try to understand some part of it.  
> The compiler makes sure in each module two unittest tests don't share  
> the same name. An example:
>
> int sqr(int x) { return 2 * x; }
>
> /// asserts that it doesn't return a negative value
> unittest(sqr) {
>     assert(sqr(10) >= 0);
>     assert(sqr(-10) >= 0);
> }

I think it's achievable through mixin templates:

mixin Unittest!("sqr", {
      assert(sqr(10) >= 0);
      assert(sqr(-10) >= 0);
});

and even doesn't poke your eyes like most mixin code:)


> 3) Each unittest error has to say the (optional) name of the unit tests  
> it is contained into. For example:
>
> test4(sqr,6): unittest failure

If Unittest template** above would be something like:

mixin template Unittest(string name, alias fun) {
     mixin("
     void "~name~"() {
         fun();
     }

     unittest {
         writeln(\"Testing "~name~"...\");
         "~name~"();
     }
     ");
}

then you would see the name of the unit test in the stack trace.


> 4) The dmd JSON output has to list all the unitttests, with their  
> optional name (because the IDE can use this information to do many  
> important things).

This is interesting. How the IDE may help knowing the unittest name?


> 5) Optional ddoc text for unittests (to allow IDEs to answer the  
> programmer the purpose of a  specific test that has failed).

When a test has failed I go to the testing code to see what's up. So I'll  
see any non-ddoc comments someone left there.


> 6a) A way to enable only unittests of a module. Because in a project  
> there are several modules, and when I work on a module I often want to  
> run only its unittests. In general it's quite useful to be able to  
> disable unittests.

... or in a package, or one folder above the package level, or ... ;)
Versioning and conditional compilation are the right tools for the job.


> 6b) A way to define groups of unittests (that cross modules too):  
> because you sometimes want to unittest for a feature spread in more than  
> one module, or you want to tell apart quick and slow unittests, to run  
> fast unittests often and slow ones only once in a while.
>
> One way to support unittest groups is to allow for tag names after the  
> unittest name that define overlapping groups:
>
> unittest(foo_name, group1_tag_name, group2_tag_name, ...) {...}

It's more of a version statement issue: can't have many versions or'ed  
together in a single condition. Then again, this is possible now:

template isVersion(string name) {
     enum bool isVersion = !is(typeof({
          mixin("version("~name~") {
                static assert(false);
          }");
      }));
}

static if (isVersion!"group1_tag_name" || isVersion!"group2_tag_name" ||  
... )
mixin Unittest!("foo_name", {...});


[snip]
> A) Serious unittest system needs a way to allow sharing of setup and  
> shutdown code for tests.

How about:

unittest {
     setUp();
     scope(exit) tearDown();
     ...
};


> Fixtures can be supported at the package, module, class and function  
> level. Setup always runs before any test (or groups of tests).

Unittests are executed in declaration order, so:

unittest { setUp(); }
unittest { ... }
unittest { ... }
unittest { tearDown(); }


In a nutshell, I agree with you that unittests would use more features but  
we should explore a (standard?) library solution first, before putting  
more metal into the language/compiler.


Tomek

** Actually, it doesn't compile now:

mixin Unittest!("my_test", {
     assert (2 + 2 == 4);
});

Error: delegate test.__dgliteral2 is a nested function and cannot be  
accessed from my_test

I'd appreciate if someone elaborated on why exactly it cannot be accessed  
 from my_test. To me, if a delegate literal is defined in the global scope,  
then, with respect to nesting, it's not much different than a normal  
function, no?