How do you test pre-/post-conditions and invariants?

[Magnus Lie Hetland]

On 2011-02-25 20:04:10 +0100, Jonathan M Davis said:

> On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:
>> Or, more generally, how do you test asserts (which is what I'm using in
>> my preconditions etc.)?
>> 
>> As far as I can see, collectException() won't collect errors, which is
>> what assert() throws -- so what's the standard way of writing unit
>> tests for preconditions that use assert? (I.e., test that they will, in
>> fact, throw when you break them.)
> 
> I think that the reality of the matter is the most of the time people _don't_
> check them. And on some level, it doesn't make sense to. It's kind of like
> asking how people test their unit tests. Unit tests are already testing 
> code. Do
> you want to be testing them on top of that? And if you do, do you test _that_
> code? Where do you stop?

I guess so. But you could say the same thing about other cases where 
you throw an exception when you detect that something is wrong -- but 
those are normally tested, right? Also, the difference here is that the 
precondition is written as a general "test", whereas my actual test 
would have specific cases.

For example, I have a test that checks that I don't add the same object 
twice to some structure, and the check involves some traversal -- code 
that could potentially be wrong. I wanted to make sure that it wasn't 
by explicitly adding the same object twice -- code (i.e., my unit test) 
that most likely could not be wrong.

But I do see your point.

[snip]
> And testing post-conditions and invariants in the manner that you're trying to
> do borders on impossible. What are you going to do, repeat the 
> post-condition or
> invariant test on the result of the function or on the state of the object that
> the function was called on after the function was called? That's just doing the
> test twice.

Right.

> You might as well just re-read the post-conditions and invariants to
> make sure that you wrote them correctly.
> 
> I do see value in testing pre-conditions if you're using exceptions rather than
> assertions (which means that you're not use in blocks). In that case, you're
> testing the API to make sure that it does what it's supposed to do. But if
> you're dealing with assertions, then it's really test code as opposed to API
> code, and I don't see the same value in testing that. You'd just be 
> testing test
> code.

OK. For the practical reason, I refer you to my explanation above. But 
I guess it's a style issue -- and I'm fine with not testing these 
things, by all means.

[snip]
> Those changes _do_ make it so that you can use collectException to
> collect an Error (though it defaults to catching Exceptions only), but 
> they also
> include assertThrown and assertNotThrown which effectively assert that the
> Exception or Error that you expected to be thrown (or not) from a particular
> expression or function call was indeed thrown (or not).
> So, you _can_ use that with AssertError to verify your pre-conditions.

OK, thanks.

> However, I would point out that catching Errors is generally a _bad_ idea.
[snip lots of useful stuff]

Thanks for educating me :D

I guess the conclusion will be that I'll focus on keeping my 
preconditions really simple. (And any utility functions I use in them 
can then get unit tests of their own instead ;)

-- 
Magnus Lie Hetland
http://hetland.org