Unit tests, asserts and contradictions in the spec

[John Colvin]

On Thursday, 7 February 2019 at 15:43:34 UTC, H. S. Teoh wrote:
> On Thu, Feb 07, 2019 at 06:51:23AM +0000, Paul Backus via 
> Digitalmars-d wrote:
>> On Thursday, 7 February 2019 at 01:04:15 UTC, H. S. Teoh wrote:
> [...]
>> > Since the nested helper function is marked nothrow, a failed 
>> > assert throwing an AssertError or UnittestAssertError may 
>> > bypass the unittest's unwinding code (the compiler assumes 
>> > runTest never throws, so never installs stack unwinding code 
>> > around it), which means the scope(exit) never triggers and 
>> > the resource is leaked.
>> 
>> I guess the implication here is that assert shouldn't count as 
>> nothrow in unit tests? That won't help non-nested helper 
>> functions, of course, but they also won't throw 
>> UnittestAssertError (in this hypothetical), so that should be 
>> okay, from a safety perspective.
>> 
>> These don't seem like insurmountable problems, but the 
>> possibility that existing code might *rely* on AssertErrors 
>> from deep in the call stack being caught by the runtime is one 
>> I hadn't considered. Since there's no way to distinguish the 
>> legitimate uses of that "feature" from the actually-broken 
>> ones, it's hard to imagine a solution that fixes the latter 
>> without breaking the former.
>
> Yeah, this is why I said this was a can o' worms. There's 
> probably a solution lurking in there somewhere, but it's 
> complicated, has messy edge cases, and I'm not sure how likely 
> it will be adopted.  It's all not so bad when you follow the 
> "assert failure == immediately terminate program" paradigm 
> strictly; but once you step outside of that, all sorts of 
> problems await.
>
> Of course, on the flip side, unittests that acquire global / 
> external resources that need cleanup, etc., are IMNSHO a code 
> smell. The code should be structured in such a way that every 
> unittest is self-contained and does not affect global state, 
> and especially not external OS state. (There's a place for such 
> tests, but they're hardly *unittests* anymore.)  If the code 
> can't be tested that way, then IMO it's suffers from bad design 
> and needs to be refactored.
>
> If all unittests were written this way, then your proposed 
> solution would totally make sense.  Unfortunately, reality is a 
> lot messier than that.  :-(
>
>
> T

A fork-based unittest runner would solve some problems without 
having to restart the process (could be expensive startup) or 
have people re-write their tests to use a new type of assert.

The process is started, static constructors are run setting up 
anything needed, the process is then forked & the tests run in 
the fork until death from success or assert, somehow 
communicating the index of the last successful test to the runner 
(let's call that tests[i]). Then if i < test.length - 1 do 
another fork and start from tests[i + 2] to skip the one that 
failed.

There are probably corner cases where you wouldn't want this 
behavior, but I can't think of one off the top of my head.