Unit tests and verifying pre/post-conditions and invariants

[Steven Schveighoffer]

On Mon, 16 Aug 2010 12:57:22 -0400, Jonathan M Davis  
<jmdavisprog at gmail.com> wrote:

> Ideally, unit tests and contracts would be simple. However, I do believe  
> that
> there is some value in verifying that you wrote you contracts correctly.
> Ideally, you could use a wrapper function/template to call the function  
> to
> verify that an AssertError was thrown, and the unit test would then be  
> simple.

I don't see why that wouldn't work.  If you really feel the need to test  
your tests.

One thing I've found invaluable, especially when writing unit tests for  
templated items, is to write the unit tests *inside* the item.  Then  
instantiate the item with all the different parameter types you want.

For example, in dcollections, all my unit tests are generic, and work with  
all value types that are integral.  Then at the end of the file I just  
have:

unittest
{
    ArrayList!ubyte al1;
    ArrayList!byte al2;
    ArrayList!ushort al3;
    ...
}

In all, I have about 8-10 different flavors of unit tests that run on each  
collection type.  I found some very obscure compiler/runtime bugs that way  
:)

> Of greater value is testing that normal exceptions are thrown when  
> they're
> supposed to - especially for bad input. Testing that is essentially the  
> same as
> testing for AssertErrors except that they're regular exceptions, so you  
> don't
> have the whole issue with Errors not getting cleaned up properly. In the  
> case of
> normal exceptions though, it is arguably part of the API (albeit not  
> part of the
> signature), while for AssertError it's a contract which isn't really  
> part of the
> API so much as verifying that your code is correct.

I whole-heartedly agree, and the code I use for this is:

bool caughtException = false;
try
{
    functionThatShouldThrow(badInput);
}
catch(SpecificException e)
{
    caughtException = true;
}
assert(caughtException);

An example in dcollections:  
http://www.dsource.org/projects/dcollections/browser/branches/d2/dcollections/HashMap.d#L757

Note another useful idiom I found: In some cases, your unit tests only  
work for certain template instantiations.  In dcollections, all my input  
to initialize the collections is done with integral literals.  So if you  
happened to instantiate an ArrayList!string, for instance, the code would  
fail to compile, because you can't add (1, 2, 3, 4, 5) to that array list.

So I define an enum bool doUnittest in each object, which looks something  
like this:

enum bool doUnittest = isIntegral!V;

Then I use the doUnittest flag to statically disable all unit tests for  
things like ArrayList of string.  You could probably define multiple  
booleans if necessary if you had unittests that did support strings.

The only annoyance is this boolean is still in the release code, but it is  
only used at compile time and it does not add to the object size, it's  
just an extra data point in the static code.

>
> In any case, I definitely see some value in testing that sort of thing.  
> You don't
> necessarily want to write unit tests for all of your contracts, but  
> sometimes it
> can be valuable. I would argue though that you generally _do_ want to  
> write code
> for normal exceptions that are thrown due to bad input (like with  
> enforce)
> because you want to guarantee the behavior of the function, and that is  
> part of
> the behavior and stays regardless of the -release flag.

I agree, you should test your normal exceptions, especially if they are  
thrown on invalid user input (i.e. deterministic).  contract asserts, I'm  
not so sure.  But I can see a reason to do that.

-Steve