Review: A new stab at a potential std.unittests [Update]

[Jacob Carlborg]

On 2010-11-27 21:36, Jonathan M Davis wrote:
> On Saturday 27 November 2010 09:00:36 Jacob Carlborg wrote:
>> On 2010-11-26 09:10, Jonathan M Davis wrote:
>>> Any more feedback on my potential std.unittests, or is looking good
>>> overall? I definitely think that it's better than when I first posted
>>> it, so the feedback thus far has definitely been helpful, and I do find
>>> these functions extremely useful in my own code.
>>>
>>> - Jonathan M Davis
>>>
>>>
>>> P.S. My most recent update for std.datetime does include the most recent
>>> version of unittests.d, so if you want to see them in more large scale
>>> use, you can look there.
>>
>> How about catching AssertErrors (if possible), collecting them and then
>> at the end of the unit test output all failed unit tests. This will
>> allow the unit tests after a failing unit test to also run.
>>
>> Feel free to have a look at my rspec inspired unit test "framework":
>> http://dsource.org/projects/orange/browser/orange/test/UnitTester.d
>
> All of my functions work essentially the same as assert does. That means that
> once an assertion fails (or one of my functions fails) in a unittest block, that
> block has failed and is done. This has been discussed before, and the general
> consensus is that this is the correct and desired behavior. It can be a bit
> annoying if you have a whole bunch of unrelated assertions in a single unittest
> block, but if you're doing anything where each assertion relies on state with an
> assertion relying on previous assertions passing, and assertions didn't really
> stop the test from running, you'd get a ton of irrelevant errors after a single
> failure.

This how it works in my unit test library:

For every test you call a testing function that takes a delegate, in 
this delegate you have your assertions. This function will call the 
delegate and catch all assert exceptions/errors and collected them for 
later outputting. Now this delegate will end execution if an exception 
is thrown and the remaining code in the delegate will not be run.

If you want your test to stop after a failing assert you put all the 
asserts into one call to the testing function. If you want your test to 
continue after a failing assert you call the testing function multiple 
times.

> My functions are intended to expand on assert, giving you better error messages
> on failures and reducing boiler plate code, not change how unit tests in D
> fundamentally work. And I don't want to introduce stuff into Phobos which changes
> how unit tests in D fundamentally work. I think that it's great as is. It's just
> that helper functions that produce better error messages than assert, and helper
> functions which reduce boiler plate code in common test cases can improve on
> assert and make for better unit tests, and I think that they would be a good
> addition to Phobos.

I guess your functions are like assert helpers and my library is at a 
higher level.

> Currently, if a unittest block fails, then none of the other unittest blocks in
> that module run, but unittest blocks in other modules do run. It's supposed to
> be changed so that all unittest blocks in a module are run even if one fails,
> but that change hasn't been made yet. Once that is made, you'll be able to split
> up your unittest blocks to more thoroughly to allow for more tests to run on a
> single failure if you'd like to. But it is by design that once you have an error
> in a unittest block, that unittest block does not continue to run.
>
> - Jonathan M Davis

-- 
/Jacob Carlborg