Extending unittests [proposal] [Proof Of Concept]

[Johannes Pfau]

Am Fri, 21 Sep 2012 11:25:10 +0200
schrieb Jacob Carlborg <doob at me.com>:

> On 2012-09-20 23:14, Jonathan M Davis wrote:
> 
> > Running more unittest blocks after a failure is similarly flawed,
> > but at least in that case, you know that had a failure earlier in
> > the module, which should then tell you that you may not be able to
> > trust further tests (but if you still run them, it's at least then
> > potentially possible to fix further failures at the same time -
> > particularly if your tests don't rely on external state). So, while
> > not necessarily a great idea, it's not as bad to run subsequent
> > unittest blocks after a failure (especially if programmers are
> > doing what they're supposed to and making their unit tests
> > independent).
> 
> I don't agree. I think that if you designed your unittests blocks so 
> they depend on other unittest blocks are equally flawed. There's a 
> reason for that most testing frameworks have "setup" and "teardown" 
> functions that are called before and after each test. With these 
> function you can restore the environment to a known state and have
> the tests keep running.
> 
> On the other hand, if there's a failure in a test, continue running
> that test would be quite bad.
> 

Well, I think we should just leave the basic unittest runner in
druntime unchanged. There are unittests in phobos which depend on that
behavior.

Other projects can use a custom test runner like Jens Mueller's dtest.

Ignoring assert in a test is not supported by this proposal. It would
need much more work and it's probably not a good idea anyway.


But when porting gdc it's quite annoying if a unit test fails because
of a compiler (codegen) error and you can't see the result of the
remaining unit tests. If unit tests are not independent, this could
cause some false positives, or crash in the worst case. But as long as
this is not the default in druntime I see no reason why we should
explicitly prevent it.
Again, the default unit test runner in druntime hasn't changed _at all_.
This just provides additional possibilities for test runners.