How do you test pre-/post-conditions and invariants?

[Jonathan M Davis]

On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:
> Or, more generally, how do you test asserts (which is what I'm using in
> my preconditions etc.)?
> 
> As far as I can see, collectException() won't collect errors, which is
> what assert() throws -- so what's the standard way of writing unit
> tests for preconditions that use assert? (I.e., test that they will, in
> fact, throw when you break them.)

I think that the reality of the matter is the most of the time people _don't_ 
check them. And on some level, it doesn't make sense to. It's kind of like 
asking how people test their unit tests. Unit tests are already testing code. Do 
you want to be testing them on top of that? And if you do, do you test _that_ 
code? Where do you stop?

Pre-conditions, post-conditions, and invariants are testing the class or struct 
that they're on, and to some extent, the code that uses them. So, if you're 
looking to test them, you're looking to test test code.

And testing post-conditions and invariants in the manner that you're trying to 
do borders on impossible. What are you going to do, repeat the post-condition or 
invariant test on the result of the function or on the state of the object that 
the function was called on after the function was called? That's just doing the 
test twice. You might as well just re-read the post-conditions and invariants to 
make sure that you wrote them correctly.

I do see value in testing pre-conditions if you're using exceptions rather than 
assertions (which means that you're not use in blocks). In that case, you're 
testing the API to make sure that it does what it's supposed to do. But if 
you're dealing with assertions, then it's really test code as opposed to API 
code, and I don't see the same value in testing that. You'd just be testing test 
code.

Now, assuming that you _do_ want to test that sort of thing (as you obviously 
want to), some new unit testing functions which would help were recently 
reviewed on the D newsgroup and voted for inclusion in Phobos. There's a pull 
request for them, but they haven't been merged in yet, and unless you use the 
development version of Phobos from git, you'll have to wait for the next release 
regardless. Those changes _do_ make it so that you can use collectException to 
collect an Error (though it defaults to catching Exceptions only), but they also 
include assertThrown and assertNotThrown which effectively assert that the 
Exception or Error that you expected to be thrown (or not) from a particular 
expression or function call was indeed thrown (or not). So, you _can_ use that 
with AssertError to verify your pre-conditions.

However, I would point out that catching Errors is generally a _bad_ idea. If 
you're careful in unit testing code, you should be okay, but _don't_ do it in 
normal code, and if you're not careful, you'll still get into trouble with unit 
testing code. Unlike with Exceptions, when an Error is thrown, scope statements 
and finally blocks do not run. I don't believe that destructors are run either. 
It's pretty much assumed that you can't recover from an Error, so no attempt to 
recover is made. Your program runs a high risk of being in an invalid state 
after an Error is thrown - AssertError included. In a unit test, if you're 
dealing with simple stuff, then you're probably okay, but if you have any code in 
there that would be affected by scope statements, finally blocks, destructors, 
etc. not running, then you don't want to be catching any Errors and then trying 
to continue.

So, functions which will help you with such testing are on their way, but they 
weren't released with the latest release of dmd, and once they have been 
released, you're going to need to be careful if you use them to test 
AssertErrors or any other kind of Error.

- Jonathan M Davis