Unit tests, asserts and contradictions in the spec

[Jacob Carlborg]

In this pull request [1] there's some discussion related to the topic of 
this post, which I would now like to make more visible.

The spec mentions that if an assertion fails the program enters invalid 
state:

"1. The first AssignExpression must evaluate to true. If it does not, an 
Assert Failure has occurred and the program enters an Invalid State." [2]

And:

"Undefined Behavior: Once in an Invalid State the behavior of the 
continuing execution of the program is undefined." [2]

Currently `assert` throws an `AssertError` if an assertion fails. This 
is mentioned in the spec to be implementation defined:

"Implementation Defined: The behavior if the first AssertExpression is 
evaluated and is false is also typically set with a compiler switch and 
may include these options:
5. throwing the AssertError exception in the D runtime library" [2]

The spec also mentions that `assert` has different semantics in 
`unittest` blocks or `in` contract:

"3. AssertExpression has different semantics if it is in a unittest or 
in contract." [2]

And:

"3. Unlike AssertExpressions used elsewhere, the assert is not assumed 
to hold, and upon assert failure the program is still in a defined 
state." [3]

Subclasses of `Error` are not supposed to be caught because they 
represent unrecoverable runtime errors. I failed to find anything about 
this in the spec but the documentation for the `Error` class specifies that:

"This represents the category of Throwable objects that are not safe to 
catch and handle. In principle, one should not catch Error objects, as 
they represent unrecoverable runtime errors. Certain runtime guarantees 
may fail to hold when these errors are thrown, making it unsafe to 
continue execution after catching them." [4]

For as long as I can remember the D runtime has been catching Throwables 
(exceptions and errors) [5]. This might be ok since the application will 
terminate shortly after. The default unit test runner will also catch 
Throwables and continue running the tests in a test fails.

So the problem is that you're not supposed to catch Errors, but you need 
to do that to implement a unit test runner that will continue after a 
failed test. If we only look at what's specified for `assert`, it should 
be fine because according to the spec if an assertion fails in a 
`unittest` block the program is still in a valid state.

The specific problem for this pull request [1] is to allocate memory in 
a function that needs to be `@nogc`, `nothrow`, `pure` and `@safe`.

[1] https://github.com/dlang/druntime/pull/2479
[2] https://dlang.org/spec/expression.html#assert_expressions
[3] https://dlang.org/spec/unittest.html
[4] https://dlang.org/phobos/object.html#.Error
[5] 
https://github.com/dlang/druntime/blob/bc940316b4cd7cf6a76e34b7396de2003867fbef/src/rt/dmain2.d#L480-L484

-- 
/Jacob Carlborg