How do you test pre-/post-conditions and invariants?

[Magnus Lie Hetland]

On 2011-02-26 13:15:58 +0100, Jonathan M Davis said:

> On Saturday 26 February 2011 03:24:15 Magnus Lie Hetland wrote:
>> OK. I had the impression that using assert() in contracts was standard,
>> also for API functions. I thought contracts fulfilled a similar sort of
>> function to assert(), in that they're removed in release code -- as
>> opposed to enforce(), for example...? I'm guessing that if I released a
>> binary version of a library, I wouldn't leave the contracts in? Or
>> perhaps I would (but, as you say, with exceptions)? Depends on the
>> situation, perhaps?
>> 
>> What kind of exceptions would be most relevant to indicate a contract
>> failure (if the contracts are made part of the API)?
> 
> Well, the biggest problem with using assertions to verify input to a 
> function is
> that if you distribute your code as a library, odds are it will be in release
> mode, and then there won't be any assertions in it.

[snip lots of stuff]

After reading your response, I first made lots of comments, but it's 
all a bit redundant. My summary is:

- You're (at times) talking about preconditions as a general concept, 
and that for public APIs, they should be enforced using exceptions.
- I've only been talking about the *language feature* of preconditions, 
i.e., in-clauses.
- We're both clear on that preconditions and asserts disappear in 
release mode, and that the two belong together, as part of your test 
scaffolding (and not as part of your public API).

Sound about right?

[snip]
> Regardless of that, however, assertions should only be used when testing the
> internal logic of your program. If code from other libraries or any other code
> which you wouldn't be looking to test calls your function, then don't use an
> assertion to verify pre-conditions. If you're using assertions, you're testing
> that the caller is correct. You're verifying that the caller is not violating
> your contract, but you're _not_ guaranteeing that the function will 
> fail if they
> violate the contract (since assertions can go away).

A very clarifying way of putting it, indeed.

As for my "testing the test code" intention, I guess (as I said) I 
actually *did* want to test the test. Not, perhaps, that it was correct 
(as discussed, it should be really simple), but to see it fail at least 
once -- a basic principle of test-driven programming. But I'll find 
other ways of doing that -- for example deliberately making the 
precondition slightly wrong at first :)

> The test for the contract  is therefore _not_ part of the API. With 
> Exceptions it _is_.

Right.

> So, what it really comes down to is whether you looking to test the 
> code which calls your function and are therefore willing to have that 
> code give you bad input and let your function process it anyway (when 
> assertions aren't compiled in) and you therefore use assertions, _or_ 
> you're looking to guarantee that your function does _not_ continue if 
> the contract is violated, and you want to _always_ error out - in which 
> case you use Exceptions.

Yep. All in all, a very useful clarification for me.

As a side note: Why isn't there a release-version of the contract 
mechanisms? I would've thought that contracts would be even more useful 
between different programmers, than just between you and yourself...?-)

That is, wouldn't the same kind of mechanism be useful for *exactly* 
the kind of exception-based input checking that you're describing as 
the alternative to contracts+asserts?

I mean, the reason to remove preconditions and asserts is primarily 
performance and not semantics (although it certainly affects semantics, 
as you've pointed out)? We have enforce() as the alternative to 
assert(); why no alternative to in/out and invariants?

[snip]
> And if you're using unit tests to test those, you're testing test code.

Sure. I've already accepted this :)

[snip]
> Still, if you start testing test code, at what point does it make sense 
> to stop?

Hm. Maybe I should write a test that tests itself?-)

More seriously: your points are well taken.

I still have a vague feeling that in-clauses are a bit different from 
out-closes, invariants and plain unit tests when it comes to the "fail 
first" approach to test-driven programming. A precondition won't fail 
because your code isn't yet functional -- it will only fail if you've 
actively written *wrong* code. But I guess that's just how it is :)

> Complicated tests of _any_ kind are a bit dangerous.
[snip]

Hm. True.

Thanks for lots of useful input!

(Still curious about the hypothetical "public API contract" 
functionality, though, and why it's non-existent.)

-- 
Magnus Lie Hetland
http://hetland.org