Extending unittests [proposal] [Proof Of Concept]

[Jens Mueller]

Jonathan M Davis wrote:
> On Thursday, September 20, 2012 22:55:23 Jens Mueller wrote:
> > You say that JUnit silently runs all unittests before the first
> > specified one, don't you?
> 
> Yes. At least, that was its behavior the last time that I used it (which was 
> admittedly a few years ago).
> 
> > If that is done silently that's indeed strange.
> 
> It could have been a quirk of their implementation, but I expect that it's to 
> avoid issues where a unit test relies on previous unit tests in the same file. 
> If your unit testing functions (or unittest blocks in the case of D) have 
> _any_ dependencies on external state, then skipping any of them affects the 
> ones that you don't skip, possibly changing the result of the unit test (be it 
> to success or failure).
> 
> Running more unittest blocks after a failure is similarly flawed, but at least 
> in that case, you know that had a failure earlier in the module, which should 
> then tell you that you may not be able to trust further tests (but if you 
> still run them, it's at least then potentially possible to fix further failures 
> at the same time - particularly if your tests don't rely on external state). 
> So, while not necessarily a great idea, it's not as bad to run subsequent 
> unittest blocks after a failure (especially if programmers are doing what 
> they're supposed to and making their unit tests independent).
> 
> However, what's truly insane IMHO is continuing to run a unittest block after 
> it's already had a failure in it. Unless you have exceedingly simplistic unit 
> tests, the failures after the first one mean pretty much _nothing_ and simply 
> clutter the results.

I sometimes have unittests like

assert(testProperty1());
assert(testProperty2());
assert(testProperty3());

And in these cases it will be useful if I got all of the assertion
failures. But you are very right that you should use it with very much
care and knowing what you do. You may even get lost not seeing the
actual problem because of so many subsequent failures.

> > When has this been merged? It must have been after v2.060 was released.
> > Because I noticed some number at the end of the unittest function names.
> > But it was not the line number.
> 
> A couple of weeks ago IIRC. I'm pretty sure that it was after 2.060 was 
> released.

I just checked.
It was merged in on Wed Sep 5 19:46:50 2012 -0700 (commit d3669f79813)
and v2.060 was released 2nd of August.
Meaning I could try calling these functions myself now that I know their
names.

Jens