Counting passed/failed unit tests

[David Gileadi]

On 10/26/11 3:45 PM, Jonathan M Davis wrote:
> On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
>> On 2011-10-26 17:40, David Gileadi wrote:
>>> On 10/25/11 4:04 AM, Jacob Carlborg wrote:
>>>> On 2011-10-24 22:08, Jonathan M Davis wrote:
>>>>> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>>>>>> I'm not sure why it just stops after the first failing unittest
>>>>>> though. What is the point of that 'failed' counter?
>>>>>
>>>>> It's a long standing issue that when one unit test fails within a
>>>>> module, no
>>>>> more within that module are run (though fortunately, a while back it
>>>>> was fixed
>>>>> so that other modules' unit tests will still run). As I recall,
>>>>> there
>>>>> had to
>>>>> be a change to the compiler to fix it, but I don't known/remember
>>>>> the
>>>>> details.
>>>>> Certainly, the issue still stands.
>>>>>
>>>>> - Jonathan M Davis
>>>>
>>>> A workaround is to catch AssertErrors, hook it up with some library
>>>> code
>>>> and you get a minimal unit test framework:
>>>> https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitT
>>>> ester.d
>>>>
>>>>
>>>>
>>>> Example of usage:
>>>>
>>>> https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d
>>>
>>> As an argument for continuing to run tests after one fails, I'm taking a
>>> TDD class and the instructor asserted that for unit tests you should
>>> generally only have one or two assertions per test method. His reasoning
>>> is that when something breaks you immediately know the extent of your
>>> breakage by counting the number of failed methods. This argument is
>>> pretty convincing to me.
>>
>> Well, in my library, if an assert error is thrown in a block (passed to
>> the "it" method), the whole block is canceled and it will continue with
>> the next block. So it's up to the user how the asserts should be laid out.
>
> It would be disastrous IMHO to continue to run a unittest block after an
> assert failed - at least in the general case. Too often further assertions
> relied on the state assured by the one that failed, so further failures just
> confuse things and give you too much data to have to sift through. As it
> stands with the built-in unit testing faciliities, you can put each assertion
> in its own unittest block if you really want each assertion to run on its own
> (though until it's fixed so that further unittest blocks within the module run
> after the first failure in that module, it wouldn't do you any good).
>
> - Jonathan M Davis

Agreed--the (rather obscure, sorry) point of my message was that having 
further unittest blocks run within the module even after one fails is a 
good idea.  Jacob's test frameworks sounds like a good effort in that 
direction, but built-in would be even better.