Counting passed/failed unit tests

[Jonathan M Davis]

On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
> On 2011-10-26 17:40, David Gileadi wrote:
> > On 10/25/11 4:04 AM, Jacob Carlborg wrote:
> >> On 2011-10-24 22:08, Jonathan M Davis wrote:
> >>> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
> >>>> I'm not sure why it just stops after the first failing unittest
> >>>> though. What is the point of that 'failed' counter?
> >>> 
> >>> It's a long standing issue that when one unit test fails within a
> >>> module, no
> >>> more within that module are run (though fortunately, a while back it
> >>> was fixed
> >>> so that other modules' unit tests will still run). As I recall,
> >>> there
> >>> had to
> >>> be a change to the compiler to fix it, but I don't known/remember
> >>> the
> >>> details.
> >>> Certainly, the issue still stands.
> >>> 
> >>> - Jonathan M Davis
> >> 
> >> A workaround is to catch AssertErrors, hook it up with some library
> >> code
> >> and you get a minimal unit test framework:
> >> https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitT
> >> ester.d
> >> 
> >> 
> >> 
> >> Example of usage:
> >> 
> >> https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d
> > 
> > As an argument for continuing to run tests after one fails, I'm taking a
> > TDD class and the instructor asserted that for unit tests you should
> > generally only have one or two assertions per test method. His reasoning
> > is that when something breaks you immediately know the extent of your
> > breakage by counting the number of failed methods. This argument is
> > pretty convincing to me.
> 
> Well, in my library, if an assert error is thrown in a block (passed to
> the "it" method), the whole block is canceled and it will continue with
> the next block. So it's up to the user how the asserts should be laid out.

It would be disastrous IMHO to continue to run a unittest block after an 
assert failed - at least in the general case. Too often further assertions 
relied on the state assured by the one that failed, so further failures just 
confuse things and give you too much data to have to sift through. As it 
stands with the built-in unit testing faciliities, you can put each assertion 
in its own unittest block if you really want each assertion to run on its own 
(though until it's fixed so that further unittest blocks within the module run 
after the first failure in that module, it wouldn't do you any good).

- Jonathan M Davis