Helper unit testing functions in Phobos (possible std.unittests)

[Jonathan M Davis]

On Saturday 06 November 2010 13:05:13 Jérôme M. Berger wrote:
> spir wrote:
> > On Sat, 6 Nov 2010 01:21:25 -0700
> > 
> > Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> >> I believe strongly that a unit test block which has a failure should end
> >> excecution. For many such tests, continuing would be utterly pointless,
> >> since each successive test relies on the last.
> > 
> > I don't understand. I can have one dozen test cases for each of one dozen
> > funcs. All 144 tests are independant. I prefere the possibility to see
> > all test errors in one go, if any. Anyway, there may be a flag
> > STOP_AT_FIRST_TEST_ERROR (or the opposite).
> 
> 	In that case, you should have a dozen dozen different test blocks,
> each for a single test case of a single function. Under Jonathan's
> proposal, when a test block fails, it would stop the current *block*
> (i.e. test case) but allow the remaining blocks to be run.

Exactly. In every other unit testing framework that I've used, a unit test is a 
function, and when that function fails, that test fails and does not continue to 
be executed. Subsequent unit test functions are run, but once a particular 
function fails, it does not continue. If you have a whole lot of unrelated 
asserts in a single function, I could see how it could be annoying to have the 
function stop on one failure, but there are plenty of cases where you have more 
complicated tests which aren't just a bunch of asserts on single function calls 
to check the result, and in those sorts of tests, the state at each assert 
relies on the state at the last assert being correct, so once one fails, all 
subsequent asserts mean nothing. So, what you do if you have a lot of indepent 
asserts, and you want more of them to run if one fails is that you break them up 
into multiple unit test functions - or in the case of D, more unit test blocks.

There was a rather long discussion on this on the Phobos list a few months back 
( you should probably start reading it here if you really want to read it: 
http://is.gd/gN1fq ) after Walter had made it so that all unit tests continued 
even after a failed assert. It was decided that that was a _bad_ idea, and 
ultimately, I believe that the decision was that each unittest block in a module 
would be run independently, but that upon any failure in a unittest block, that 
block would terminate. However, there are changes that must be made in the 
compiler to make it possible to make it so that subsequent unittest blocks in a 
module run after a failure in one, so at the moment, if you get a failure in a 
unittest block, none of the other unittest blocks in that module run (though the 
unittest blocks in other modules still run). The change should be made 
eventually, however. Actually, I should probably make sure that it's in bugzilla 
so that it isn't forgotten.

- Jonathan M Davis