[phobos] Silent failure of std.container unittests

[Michel Fortin]

Le 2010-07-14 à 22:42, Andrei Alexandrescu a écrit :

> On 07/14/2010 08:58 PM, Walter Bright wrote:
>> 
>> 
>> Sean Kelly wrote:
>>> 
>>> 
>>> How about this... since unit tests are run after static ctors are run,
>>> the behavior of whether to throw an AssertError or report and continue
>>> can be a run-time configurable option.
>> 
>> Frankly, I don't think a configurable option is a good idea. Druntime
>> has a lot of user configurability, and as far as I can tell, absolutely
>> none of it has ever been used. It just adds complexity, both to the code
>> and the documentation.
> 
> I agree. But also simple does not mean crappy. I think there's too much talk around a simple matter:
> 
> ======
> Assertion failure should abort the current unittest block.
> ======
> 
> Do we all agree about the above? We keep on discussing how the usefulness and informativeness of assertion failures falls off a cliff after the first failure, and I can't seem to hear loud and clear that we all want this same thing.
> 
> So we've been through these steps:
> 
> 1. Crappy: failure to assert causes the program to abort.
> 
> 2. Awful: assert is hacked inside unittests to essentially be writeln, all unittests run regardless. Programs claims success upon exit.
> 
> 3. Mediocre: assert continues to be hacked inside unittests to essentially be writeln + set a flag. Programs exit with errors if the flag was set.
> 
> NOT YET:
> 
> 4. DESIRED: assert is NOT hacked, any failing assert ends the current unittest, the failure message is printed, execution continues with the next unittest, program ends with error code if at least one assert failed, everybody's happy.
> 
> I'm glad we made progress from 2 to 3. Could you all please share your opinion about the delta between 3 and 4?

In my order of preference: 4, 1, 3, 2. I think 4 is what most people want in most situations, and it also fit much better with the expected behaviour of assert (which should throw). It's also what most unit test frameworks do.

I'm currently working with a unit test framework that happens to do 3 (SenTestingKit, Objective-C) and it's frustrating that when I have only two tests that fails I get 14 errors, only two of them meaningful (got that just a few hours ago). Fortunately for me, I use Xcode to visualize the results which groups those errors by unit test, making it easier to see how many and which tests are failing and navigate around: in this situation it might make some sense to allow tests to continue after a failed assertion even though the utility is limited. But for this kind of visualization, you'd need a unit tests delimiter in the output (and ideally unit test names), and you also need to subvert 'assert' not to throw, both of which do not fit well with D. (And you need an IDE to parse and display the results better.)

So I think 4 is the best option. 1 is second best. 3 is mostly the same as one, except you have a lot of garbage (meaningless errors) following the first error. 4 is downright silly (for a default behaviour). I can see some situation where each of these options can be useful, but 4 is the one that makes most sense most of the time and fits .

-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/