[phobos] Silent failure of std.container unittests

[Jonathan M Davis]

On Wednesday, July 14, 2010 16:30:58 Walter Bright wrote:
> Jonathan M Davis wrote:
> > On the other hand, never halting on assertion failures makes for a lot
> > of cruft in your results.
> 
> I don't really understand why that's a problem, or causes extra work.
> 
> But I can see that having to divide up the tests into multiple unittest
> blocks adds extra work.
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Well, I find it perfectly normal having to divide up the unit tests since every 
unit testing framework that I've used divides up unit tests into functions where 
each function is a test that succeeds or fails. There may be many assertions in 
a particular function, but once one fails, that function ceases to run 
(generally because an exception was thrown which is then caught by the 
framework). So, it actually makes more sense to me to divide up tests into unit 
test blocks. True, if you're testing a set of small but unrelated stuff, it could 
make sense to put them all together in one test, but if you're testing blocks as 
a whole, then you have the choice of whether to break them up or not just like 
in another framework, you'd have the choice of whether you have a single 
function or multiple functions.

The cruft is a problem because it makes it harder to find the errors you actually 
care about. Unless you're checking a bunch of small, unrelated things, any 
assertion failure after the initial error is likely to be totally erroneous 
(since it was testing invalid state to begin with), so all of those errors just 
clutter the output, making it harder to find where one unittest block's tests end 
and where another begins (since it'll be the beginning of the next block where 
you'll start caring about errors again).

It's definitely better to run all tests in a block regardless of assertion 
failures than it is to stop running all of the unittests in a program or module 
because one assertion failed. However, I do think that it's better to not 
continue to run a particular unittest block after an assertion fails because 
then it's likely to be testing invalid state and report a bunch of errors which 
are irrelevant.

If you look at each unittest block as being a unit test testing a set of 
functionality rather than a particular assertion being the test, then it's just 
plain weird to continue to run the tests after the first assertion failed. The 
test already failed.

However, if you're viewing each assertion as a test with the idea that there may 
be many unrelated assertions together, then I can see why it might be desirable 
to have the block continue to be executed (since splitting them up would mean a 
lot of unittest blocks).

In any case, I do think that continuing to execute a unittest block after a 
failure is odd, and that it will result in a lot of unnecessary cruft, so I'd 
prefer that that didn't happen. However, it's definitely better to have the test 
continue than to stop them all after a single assertion failure. I suppose that 
the ideal solution would be to allow for both by having some way of indicating 
whether you wanted the test to continue after a particular assertion failure, 
but that would likely require multiple assertion types, which we don't have 
right now (at least not that the programmer can choose). Maybe if asserts threw 
normally (making it so that the first assertion failure in the unittest block 
resulted in that block ceasing execution, followed by the next unittest block 
being executed) but we had a library function of some kind which simply printed 
an error rather than throwing that would allow for the best of both worlds (like 
assert_nothrow() or something similar).

- Jonathan M Davis