Unit tests and verifying pre/post-conditions and invariants

[Jonathan M Davis]

On Monday, August 16, 2010 05:43:08 Steven Schveighoffer wrote:
> On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis
> 
> <jmdavisprog at gmail.com> wrote:
> > Is there a standard and/or acceptable way to make sure that
> > pre-conditions,
> > post-conditions, or invariants _fail_ when running unit tests? That is,
> > lets say
> > I had a function like this
> > 
> > void func(int x)
> > in
> > {
> > 
> >     assert(x < 8);
> > 
> > }
> > body
> > {
> > 
> >   //...
> > 
> > }
> > 
> > 
> > and I wanted to test to make sure that func() couldn't be called with
> > any int
> > greater or equal to 8, what would I do?
> 
> Hm... unit testing your unit tests :)
> 
> input contracts and unit tests are supposed to be simple, provable code so
> you don't have to test them.  The above function is obviously a simple
> example, you don't really need to unit test it (right?), so what would a
> complicated in contract look like?
> 
> It's also a good idea to avoid complex expressions in unit tests.  If you
> have a complex expression, split it out into several lines to avoid having
> to work through the logic in your head.  Performance/minimal LOC is not a
> goal you need in unit tests/contracts.
> 
> -Steve

Ideally, unit tests and contracts would be simple. However, I do believe that 
there is some value in verifying that you wrote you contracts correctly. 
Ideally, you could use a wrapper function/template to call the function to 
verify that an AssertError was thrown, and the unit test would then be simple.

Of greater value is testing that normal exceptions are thrown when they're 
supposed to - especially for bad input. Testing that is essentially the same as 
testing for AssertErrors except that they're regular exceptions, so you don't 
have the whole issue with Errors not getting cleaned up properly. In the case of 
normal exceptions though, it is arguably part of the API (albeit not part of the 
signature), while for AssertError it's a contract which isn't really part of the 
API so much as verifying that your code is correct.

In any case, I definitely see some value in testing that sort of thing. You don't 
necessarily want to write unit tests for all of your contracts, but sometimes it 
can be valuable. I would argue though that you generally _do_ want to write code 
for normal exceptions that are thrown due to bad input (like with enforce) 
because you want to guarantee the behavior of the function, and that is part of 
the behavior and stays regardless of the -release flag.

- Jonathan M Davis