Unit tests, asserts and contradictions in the spec

[H. S. Teoh]

On Thu, Feb 07, 2019 at 06:51:23AM +0000, Paul Backus via Digitalmars-d wrote:
> On Thursday, 7 February 2019 at 01:04:15 UTC, H. S. Teoh wrote:
[...]
> > Since the nested helper function is marked nothrow, a failed assert
> > throwing an AssertError or UnittestAssertError may bypass the
> > unittest's unwinding code (the compiler assumes runTest never
> > throws, so never installs stack unwinding code around it), which
> > means the scope(exit) never triggers and the resource is leaked.
> 
> I guess the implication here is that assert shouldn't count as nothrow
> in unit tests? That won't help non-nested helper functions, of course,
> but they also won't throw UnittestAssertError (in this hypothetical),
> so that should be okay, from a safety perspective.
> 
> These don't seem like insurmountable problems, but the possibility
> that existing code might *rely* on AssertErrors from deep in the call
> stack being caught by the runtime is one I hadn't considered. Since
> there's no way to distinguish the legitimate uses of that "feature"
> from the actually-broken ones, it's hard to imagine a solution that
> fixes the latter without breaking the former.

Yeah, this is why I said this was a can o' worms. There's probably a
solution lurking in there somewhere, but it's complicated, has messy
edge cases, and I'm not sure how likely it will be adopted.  It's all
not so bad when you follow the "assert failure == immediately terminate
program" paradigm strictly; but once you step outside of that, all sorts
of problems await.

Of course, on the flip side, unittests that acquire global / external
resources that need cleanup, etc., are IMNSHO a code smell. The code
should be structured in such a way that every unittest is self-contained
and does not affect global state, and especially not external OS state.
(There's a place for such tests, but they're hardly *unittests*
anymore.)  If the code can't be tested that way, then IMO it's suffers
from bad design and needs to be refactored.

If all unittests were written this way, then your proposed solution
would totally make sense.  Unfortunately, reality is a lot messier than
that.  :-(


T

-- 
PNP = Plug 'N' Pray