How do you test pre-/post-conditions and invariants?

[Jonathan M Davis]

On Saturday 26 February 2011 03:24:15 Magnus Lie Hetland wrote:
> On 2011-02-26 01:20:49 +0100, Jonathan M Davis said:
> > So, using such assertions makes good sense when you control
> > both the caller and the callee and it's something that should never
> > happen.
> 
> Yeah, in my case that's what's going on. I'll only be using the
> contracts during testing anyway, and remove them with -release in the
> code that's actually to be used. (The code is part of some benchmarking
> experiments, so I'd rather not have any kind of checking like that when
> running those.)
> 
> > However, if you don't necessarily control the caller or if it's something
> > that _could_ happen at runtime (even if it shouldn't), then an exception
> > makes a lot more sense.
> 
> OK. I had the impression that using assert() in contracts was standard,
> also for API functions. I thought contracts fulfilled a similar sort of
> function to assert(), in that they're removed in release code -- as
> opposed to enforce(), for example...? I'm guessing that if I released a
> binary version of a library, I wouldn't leave the contracts in? Or
> perhaps I would (but, as you say, with exceptions)? Depends on the
> situation, perhaps?
> 
> What kind of exceptions would be most relevant to indicate a contract
> failure (if the contracts are made part of the API)?

Well, the biggest problem with using assertions to verify input to a function is 
that if you distribute your code as a library, odds are it will be in release 
mode, and then there won't be any assertions in it. In that case, I believe that 
template functions will still end up with the assertions in it when the user of 
your library compiles with assertions enabled, since template code is not 
generated until it's instantiated, but none of the other assertions will work. 
So, assertions for public APIs really don't work very well. On top of that, even 
if assertions _are_ enabled (either because it's a templated function or they're 
actually using a non-release version of your library), then an assertion failure 
makes it look like _your_ code is wrong rather than theirs.

Regardless, if you're using an assertion, what you're doing is requiring that 
the input meet some sort of pre-conditions and you're testing that the caller's 
code to verify that it meets those conditions. If you use an exception, then 
it's perfectly legitimate for a caller to give input which violates your pre-
conditions, but then the caller has to deal with it. In some cases, you actually 
_need_ to do it that way simply because the input could reasonably be invalid at 
runtime, and which point it _needs_ to be checked at runtime and have the error 
reported without killing the program - which means that you need an exception.

Personally, I only ever use assertions for pre-conditions if I'm controlling 
both the caller and the callee and I really expect that they will _never_ fail. 
It's test code plain and simple. In pretty much all of the cases, I use 
exceptions. Assertions are purely for catching logic errors in code.

Now, if you're throwing an exception from a function due to a pre-condition 
failure, then the type that you throw depends entirely on what you're doing. In 
Phobos, it tends to depend on what module it's in. std.datetime throws 
DateTimeExceptions. std.file throws FileExceptions. In other cases, it's more 
specific. e.g. std.typecons.conv throws ConvExceptions. What type of Exception a 
function throws is entirely up to you. It could be a plain old Exception if 
that's what you want. It's just that if it's a specific type of Exception than 
code can catch that specific type and handle it differently than it might handle a 
generic Exception.

> > I tend to view exceptions as part of the API and think that they should
> > be tested. Assertions, on the other hand, aren't really part of the API,
> > since they
> > go away in release mode, and I therefore view them as test code. They're
> > verifying that your logic is correct.
> 
> Exactly. The same, of course, applies to contracts -- which is why I'm
> a bit confused by your suggestion to use exceptions in them.
> 
> Or perhaps I'm misreading you completely, and you're only suggesting
> that I use code paths that throw exceptions in the function body
> itself, e.g., with enforce(foo, exception) (which would make sense to
> me)?

Never throw exceptions form in, out, or invariant blocks. They'll just go away 
in release mode. Only use assertions in there. So, if you're going to throw an 
Exception, throw it from the function body or from some other function that gets 
called by that function.

Regardless of that, however, assertions should only be used when testing the 
internal logic of your program. If code from other libraries or any other code 
which you wouldn't be looking to test calls your function, then don't use an 
assertion to verify pre-conditions. If you're using assertions, you're testing 
that the caller is correct. You're verifying that the caller is not violating 
your contract, but you're _not_ guaranteeing that the function will fail if they 
violate the contract (since assertions can go away). The test for the contract 
is therefore _not_ part of the API. With Exceptions it _is_. So, what it really 
comes down to is whether you looking to test the code which calls your function 
and are therefore willing to have that code give you bad input and let your 
function process it anyway (when assertions aren't compiled in) and you 
therefore use assertions, _or_ you're looking to guarantee that your function 
does _not_ continue if the contract is violated, and you want to _always_ error 
out - in which case you use Exceptions. If you're dealing with any kind of 
public API, Exceptions are going to tend to be the correct way to go. If it's 
just your code and you control the caller and want to test it, then assertions 
may be the correct way to go.

In any case, don't throw exceptions from in, out, or invariant blocks. They 
definitely go in the function bodies.

> > So, on some level, it is indeed a stylistic thing, but where you choose
> > to use exceptions and where you choose to use assertions can have a big
> > effect on code that uses your code.
> 
> Sure thing. It just seems to me that contracts and assertions go well
> together, and have the same function, of testing your program logic?
> 
> I guess the driving force of my original query was the old "first, see
> your test fail" idea of test-driven programming. If I just slap a
> precondition on some code, it won't fail because things aren't
> implemented properly yet (as a normal unit test would) -- I'd actively
> have to implement a call to it *improperly*. It just seemed naturally
> to me to do that as part of the test code, rather than a one-off thing
> in the main code.
> 
> However, I could always add a call to my unit test, run it, and see it
> crash -- and then comment it out. Doesn't seem like the prettiest way
> to handle things, but it's OK, I guess together with the idea of making
> the contracts super-simple (and to test any functionality they use
> separately).

That makes sense except that what you're really testing when you assert pre-
conditions is the caller code, not the function that they're in. And when you do 
unit tests, you're normally testing the functions that you call. You throw the 
assertions in there to make sure that the caller code doesn't violate the pre-
conditions, so it's test code. And if you're using unit tests to test those, 
you're testing test code. There's nothing stopping you from doing it, but it's 
already undefined behavior per DbC when a pre-condition is violated, and if 
you're then testing those asserted pre-conditions with unit tests, you're 
testing to make sure that the behavior _is_ defined (that it throws an 
AssertError).

I suppose that while from the standpoint of principle it's just plain weird if 
not outright wrong to test that your assertions which test your pre-conditions 
actually throw AssertErrors when they're supposed to and don't when they're not, 
it _does_ have some practical benefit. Still, if you start testing test code, at 
what point does it make sense to stop?

Regardless, it's totally up to you when and where you use assertions or 
exceptions. But in general, assertions are really test code and shouldn't be 
considered part of the API (so testing them as part of the API like you're 
looking to do is just plain weird), whereas Exceptions most definitely _are_ part 
of the API. And from a perfectly practical standpoint, as soon as your code ends 
up in a library, assertions are generally useless anyway, because they likely 
won't be enabled, and anyone using your library won't ever see them even if they 
violate your pre-conditions on every call that they make.

> [snip]
> 
> >> I guess the conclusion will be that I'll focus on keeping my
> >> preconditions really simple. (And any utility functions I use in them
> >> can then get unit tests of their own instead ;)
> > 
> > That's probably a good way to handle it .
> 
> OK, good :)

Complicated tests of _any_ kind are a bit dangerous. If your test code (be it 
assertions in DbC or in unit tests or wherever) should be simple enough that 
you're unlikely to screw it up. You don't want to have much risk of getting your 
test code wrong. It makes it far too likely to miss bugs, and it makes it much 
harder to determine whether a test failure is due to the code being tested being 
bad or due to the test being bad.

- Jonathan M Davis