[phobos] Silent failure of std.container unittests

[Sean Kelly]

On Jul 14, 2010, at 7:42 PM, Andrei Alexandrescu wrote:

> On 07/14/2010 08:58 PM, Walter Bright wrote:
>> 
>> 
>> Sean Kelly wrote:
>>> 
>>> 
>>> How about this... since unit tests are run after static ctors are run,
>>> the behavior of whether to throw an AssertError or report and continue
>>> can be a run-time configurable option.
>> 
>> Frankly, I don't think a configurable option is a good idea. Druntime
>> has a lot of user configurability, and as far as I can tell, absolutely
>> none of it has ever been used. It just adds complexity, both to the code
>> and the documentation.
> 
> I agree. But also simple does not mean crappy. I think there's too much talk around a simple matter:
> 
> ======
> Assertion failure should abort the current unittest block.
> ======
> 
> Do we all agree about the above?

Yes.

> We keep on discussing how the usefulness and informativeness of assertion failures falls off a cliff after the first failure, and I can't seem to hear loud and clear that we all want this same thing.
> 
> So we've been through these steps:
> 
> 1. Crappy: failure to assert causes the program to abort.
> 
> 2. Awful: assert is hacked inside unittests to essentially be writeln, all unittests run regardless. Programs claims success upon exit.
> 
> 3. Mediocre: assert continues to be hacked inside unittests to essentially be writeln + set a flag. Programs exit with errors if the flag was set.
> 
> NOT YET:
> 
> 4. DESIRED: assert is NOT hacked, any failing assert ends the current unittest, the failure message is printed, execution continues with the next unittest, program ends with error code if at least one assert failed, everybody's happy.
> 
> I'm glad we made progress from 2 to 3. Could you all please share your opinion about the delta between 3 and 4?

That's what I want.  Most is a trivial library change which I've already made and have been waiting to commit, the missing piece is the unittest block-level granularity for test recovery.  That requires a compiler change to actually expose unit tests at that level of granularity.  My vacation ends tomorrow and I'll see if I can figure out how to patch this into the compiler.  I really haven't spent the time to understand how it works yet.