Counting passed/failed unit tests

[Jonathan M Davis]

On Wednesday, October 26, 2011 08:40 David Gileadi wrote:
> On 10/25/11 4:04 AM, Jacob Carlborg wrote:
> > On 2011-10-24 22:08, Jonathan M Davis wrote:
> >> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
> >>> I'm not sure why it just stops after the first failing unittest
> >>> though. What is the point of that 'failed' counter?
> >> 
> >> It's a long standing issue that when one unit test fails within a
> >> module, no
> >> more within that module are run (though fortunately, a while back it
> >> was fixed
> >> so that other modules' unit tests will still run). As I recall, there
> >> had to
> >> be a change to the compiler to fix it, but I don't known/remember the
> >> details.
> >> Certainly, the issue still stands.
> >> 
> >> - Jonathan M Davis
> > 
> > A workaround is to catch AssertErrors, hook it up with some library code
> > and you get a minimal unit test framework:
> > https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTest
> > er.d
> > 
> > 
> > Example of usage:
> > 
> > https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d
> 
> As an argument for continuing to run tests after one fails, I'm taking a
> TDD class and the instructor asserted that for unit tests you should
> generally only have one or two assertions per test method. His
> reasoning is that when something breaks you immediately know the extent
> of your breakage by counting the number of failed methods. This
> argument is pretty convincing to me.

The value of that would depend on what exactly the tests are doing - 
particularly if tests require some setup - but I could see how it would be 
valuable to essentially list all of the failed assertions rather than just 
failed unittest blocks. However, I don't think that I'd ever do it that way 
simply because the code clutter would be far too great. Even if your unit 
tests consist simply of a bunch of one-liners which are just assertions 
without any setup, having a unittest block per assertion is going to create a 
lot of extra code, which will seriously clutter the file. Now, you have the 
choice to do it either way, which is great, but I don't think that I'd want to 
be leaning towards one assertion per unittest block. I'd generally favor one 
unittest block per function, or at least group of related tests.

- Jonathan M Davis