[phobos] Silent failure of std.container unittests

[Jonathan M Davis]

On Wednesday 14 July 2010 19:42:06 Andrei Alexandrescu wrote:
> On 07/14/2010 08:58 PM, Walter Bright wrote:
> > Sean Kelly wrote:
> >> How about this... since unit tests are run after static ctors are run,
> >> the behavior of whether to throw an AssertError or report and continue
> >> can be a run-time configurable option.
> > 
> > Frankly, I don't think a configurable option is a good idea. Druntime
> > has a lot of user configurability, and as far as I can tell, absolutely
> > none of it has ever been used. It just adds complexity, both to the code
> > and the documentation.
> 
> I agree. But also simple does not mean crappy. I think there's too much
> talk around a simple matter:
> 
> ======
> Assertion failure should abort the current unittest block.
> ======
> 
> Do we all agree about the above?

Yes.

> We keep on discussing how the
> usefulness and informativeness of assertion failures falls off a cliff
> after the first failure, and I can't seem to hear loud and clear that we
> all want this same thing.
> 
> So we've been through these steps:
> 
> 1. Crappy: failure to assert causes the program to abort.
> 
> 2. Awful: assert is hacked inside unittests to essentially be writeln,
> all unittests run regardless. Programs claims success upon exit.
> 
> 3. Mediocre: assert continues to be hacked inside unittests to
> essentially be writeln + set a flag. Programs exit with errors if the
> flag was set.
> 
> NOT YET:
> 
> 4. DESIRED: assert is NOT hacked, any failing assert ends the current
> unittest, the failure message is printed, execution continues with the
> next unittest, program ends with error code if at least one assert
> failed, everybody's happy.
> 
> I'm glad we made progress from 2 to 3. Could you all please share your
> opinion about the delta between 3 and 4?

4 is far more desirable. 3 makes assert function abnormally in comparison to 
normal code and will likely confuse people. It's also likely to result in lots 
of extraneous failures. 4 is what all the unit test frameworks I have used do, 
so I'm inclined to think that it's more standard. The only downside that I see 
to 4 is that if you want to do a lot of little tests and have them all run 
regardless of previous failures, you need a lot of unittest blocks. But in every 
other unit test framework, you'd need a full-on function for every such test 
(possibly including prototypes in a header file and the like), so it's no worse 
than what I'm used to.

I much prefer 3 over 2, and it's probably better than 1 (depending on how many 
tests you have failing), but 4 is, in my opinion, hands down better than 3. 3's 
behaviour is just plain odd - better than 2's behavior - but odd, and likely 
confusing to anyone new to D.

- Jonathan M Davis