How do you test pre-/post-conditions and invariants?

[Jonathan M Davis]

On Saturday 26 February 2011 07:03:29 Magnus Lie Hetland wrote:
> On 2011-02-26 13:15:58 +0100, Jonathan M Davis said:
> > On Saturday 26 February 2011 03:24:15 Magnus Lie Hetland wrote:
> >> OK. I had the impression that using assert() in contracts was standard,
> >> also for API functions. I thought contracts fulfilled a similar sort of
> >> function to assert(), in that they're removed in release code -- as
> >> opposed to enforce(), for example...? I'm guessing that if I released a
> >> binary version of a library, I wouldn't leave the contracts in? Or
> >> perhaps I would (but, as you say, with exceptions)? Depends on the
> >> situation, perhaps?
> >> 
> >> What kind of exceptions would be most relevant to indicate a contract
> >> failure (if the contracts are made part of the API)?
> > 
> > Well, the biggest problem with using assertions to verify input to a
> > function is
> > that if you distribute your code as a library, odds are it will be in
> > release mode, and then there won't be any assertions in it.
> 
> [snip lots of stuff]
> 
> After reading your response, I first made lots of comments, but it's
> all a bit redundant. My summary is:
> 
> - You're (at times) talking about preconditions as a general concept,
> and that for public APIs, they should be enforced using exceptions.
> - I've only been talking about the *language feature* of preconditions,
> i.e., in-clauses.
> - We're both clear on that preconditions and asserts disappear in
> release mode, and that the two belong together, as part of your test
> scaffolding (and not as part of your public API).
> 
> Sound about right?

Yeah.

> [snip]
> 
> > Regardless of that, however, assertions should only be used when testing
> > the internal logic of your program. If code from other libraries or any
> > other code which you wouldn't be looking to test calls your function,
> > then don't use an assertion to verify pre-conditions. If you're using
> > assertions, you're testing that the caller is correct. You're verifying
> > that the caller is not violating your contract, but you're _not_
> > guaranteeing that the function will fail if they
> > violate the contract (since assertions can go away).
> 
> A very clarifying way of putting it, indeed.
> 
> As for my "testing the test code" intention, I guess (as I said) I
> actually *did* want to test the test. Not, perhaps, that it was correct
> (as discussed, it should be really simple), but to see it fail at least
> once -- a basic principle of test-driven programming. But I'll find
> other ways of doing that -- for example deliberately making the
> precondition slightly wrong at first :)
>
> > The test for the contract  is therefore _not_ part of the API. With
> > Exceptions it _is_.
> 
> Right.

If you really want to test the test code, then test the test code. But even in 
test driven development, you _wouldn't_ be testing that the function fails when 
given values which violate its pre-conditions. That is _undefined_ behavior and 
arguably doesn't matter. The caller violated the pre-condition. They get what 
they get. The behavior is completely undefined at that point. You just put in 
assertions to test pre-conditions so that you can find bugs in the calling code. 
As soon as you're testing that an AssertError is thrown, you're testing behavior 
that DbC considers undefined. DbC says that if you give a function input that 
does not violate its pre-conditions, the function will give you output which 
does not violate its post-conditions. It says nothing about what happens when 
you give it input which violates your pre-conditions. All bets are off at that 
point. You violated the contract.

So, testing your pre-conditions with assertions is simply testing that the 
caller code is correct. The function itself doesn't care whether the input was 
correct or not. It's only obligated to return valid values when the contract is 
kept. If the caller violates the contract, then tough luck for it.

The difference with exceptions is that you're _requiring_ that the caller give 
you correct input and erroring out if it doesn't.

Regardless, if you really want to test that your pre-condition checks are 
correct, then just test them. I'd argue against it because you're testing test 
code, and that shouldn't be necessary, but if you're going to test that your 
assertions work correctly, then test them right.

> > So, what it really comes down to is whether you looking to test the
> > code which calls your function and are therefore willing to have that
> > code give you bad input and let your function process it anyway (when
> > assertions aren't compiled in) and you therefore use assertions, _or_
> > you're looking to guarantee that your function does _not_ continue if
> > the contract is violated, and you want to _always_ error out - in which
> > case you use Exceptions.
> 
> Yep. All in all, a very useful clarification for me.
> 
> As a side note: Why isn't there a release-version of the contract
> mechanisms? I would've thought that contracts would be even more useful
> between different programmers, than just between you and yourself...?-)
> 
> That is, wouldn't the same kind of mechanism be useful for *exactly*
> the kind of exception-based input checking that you're describing as
> the alternative to contracts+asserts?
> 
> I mean, the reason to remove preconditions and asserts is primarily
> performance and not semantics (although it certainly affects semantics,
> as you've pointed out)? We have enforce() as the alternative to
> assert(); why no alternative to in/out and invariants?

Contracts are meant for testing _your_ code. Yes, in blocks are testing the 
caller's code, but you do all of that because you want to verify that _your_ 
code is correct. If other libraries or programs are using your code, then they 
should be able to assume that your code is correct and not need to have all of 
its internal checks running, slowing it down.

If a caller violates your pre-conditions, then they're violating the contract, 
and your code has no obligation at that point to give them anything in the way 
of error messages or correct output. DbC is specifically saying that they're 
_required_ to give you correct input. The fact that they didn't is their fault 
and their problem. Assertions test that your code follows the contract. Whether 
other code does or not is its problem.

Now, if you want to actually have your code error out when given bad input, then 
you use exceptions (enforce is one way to do that). But all you'd be using them 
for is input. You're not going to throw an exception on bad output, because 
that's a bug in _your_ code, not theirs (the same goes for invariants). And 
since exceptions are then part of the function's normal operation rather than 
testing, it makes no sense to put them in an in block. What does an in block buy 
you at that point, even if it does stick around? Just put the checks at the top 
of your function and throw if they fail. And honestly, there are plenty of times 
when such checks actually need to go deeper in the function anyway, because you 
have to process some of the input before you know that it's invalid.

> [snip]
> 
> > And if you're using unit tests to test those, you're testing test code.
> 
> Sure. I've already accepted this :)
> 
> [snip]
> 
> > Still, if you start testing test code, at what point does it make sense
> > to stop?
> 
> Hm. Maybe I should write a test that tests itself?-)
> 
> More seriously: your points are well taken.
> 
> I still have a vague feeling that in-clauses are a bit different from
> out-closes, invariants and plain unit tests when it comes to the "fail
> first" approach to test-driven programming. A precondition won't fail
> because your code isn't yet functional -- it will only fail if you've
> actively written *wrong* code. But I guess that's just how it is :)

in blocks are definitely different in that they test the caller's code, but if all 
of the code involved is _your_ code, then that does make some sense. It's when 
it's part of an API that it doesn't.

> > Complicated tests of _any_ kind are a bit dangerous.
> 
> [snip]
> 
> Hm. True.
> 
> Thanks for lots of useful input!
> 
> (Still curious about the hypothetical "public API contract"
> functionality, though, and why it's non-existent.)

The contracts are there. They're what is agreed upon. What's not there is 
_testing_ that those contracts aren't violated. And since assertions are really 
for testing the internal logic of your own code, it really doesn't make sense to 
be throw AssertErrors at 3rd party code that calls yours. It just slows your 
code down having to do the checks makes it look like your code is wrong when 
they give you bad input.

- Jonathan M Davis