Unit tests, asserts and contradictions in the spec

[H. S. Teoh]

On Wed, Feb 06, 2019 at 10:02:05PM +0100, Jacob Carlborg via Digitalmars-d wrote:
[...]
> The spec mentions that if an assertion fails the program enters
> invalid state:
> 
> "1. The first AssignExpression must evaluate to true. If it does not,
> an Assert Failure has occurred and the program enters an Invalid
> State." [2]
> 
> And:
> 
> "Undefined Behavior: Once in an Invalid State the behavior of the
> continuing execution of the program is undefined." [2]
[...]
> "3. AssertExpression has different semantics if it is in a unittest or
> in contract." [2]
> 
> And:
> 
> "3. Unlike AssertExpressions used elsewhere, the assert is not assumed
> to hold, and upon assert failure the program is still in a defined
> state." [3]
> 
> Subclasses of `Error` are not supposed to be caught because they
> represent unrecoverable runtime errors. I failed to find anything
> about this in the spec but the documentation for the `Error` class
> specifies that:
> 
> "This represents the category of Throwable objects that are not safe
> to catch and handle. In principle, one should not catch Error objects,
> as they represent unrecoverable runtime errors. Certain runtime
> guarantees may fail to hold when these errors are thrown, making it
> unsafe to continue execution after catching them." [4]
[...]

This is a big can o' worms I'm not sure we're ready to deal with just
yet.  Generally, everywhere except in unittests we expect assert
failures to basically terminate the program instantly, because it's in
undefined state, which is UB by definition in the spec, and UB is bad
because there's no guarantee any further operations will actually do
what you think it does.  E.g., the assert failure could have been caused
by a hacker triggering an invalid state and inserting shellcode into
memory somewhere that the code is now about to access.  Continuing to
run means the hacker will probably compromise your system.

In unittests, however, we've had the convention of using asserts for
test failures since the early days.  Generally, UB caused by asserts in
this context probably wouldn't have the same risk as above, but it does
bring into question the wisdom of continuing to run after a unittest has
failed.  The trouble is that the assert may have happened deep inside
the call stack, and since it throws AssertError, that means stack
unwinding code is potentially never run (or there is no unwinding code
to begin with because the assert is inside a nothrow function -- it's
allowed to throw non-Exception Errors from nothrow functions because
it's assumed to be fatal).  So you may end up with unreleased resources
and cleanups that ought to have happened but never did.  In the rare
case where this matters, it can cause problems with running unittests.

Worst of all, however, is the fact that contract failures also use
assert to signal the failure of a contract.  One might think this makes
sense -- because a contract failure means the program has entered
invalid state, so we're ready to abort anyway.  However, there's one
nasty catch: overriding a base class function with an in-contract is
defined by the spec to *loosen* the contract. I.e., the base class
contract may fail but it's not assumed to be invalid state / UB land,
because the derived class is allowed to loosen the contract.

But this means that if the contract failed because of some deeply-nested
assert while evaluating the contract. E.g., the contract calls some
function which in turns calls a bunch of stuff, and somewhere in there
an assert fails. AssertError is thrown, possibly through nothrow
boundaries and causing non-cleanup of various resources and states,
before it reaches the base class contract that then forwards the problem
to the derived class contract.  The derived contract says everything
checks out, so we continue running.  In spite of the resource leak
caused by the implicit thrown AssertError.


T

-- 
Notwithstanding the eloquent discontent that you have just respectfully expressed at length against my verbal capabilities, I am afraid that I must unfortunately bring it to your attention that I am, in fact, NOT verbose.