[phobos] Silent failure of std.container unittests

[Steve Schveighoffer]

----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Wed, July 14, 2010 10:42:06 PM
> Subject: Re: [phobos] Silent failure of std.container unittests
> 
> On 07/14/2010 08:58 PM, Walter Bright wrote:
> > 
> > 
> > Sean  Kelly wrote:
> >> 
> >> 
> >> How about this... since unit  tests are run after static ctors are run,
> >> the behavior of whether to  throw an AssertError or report and continue
> >> can be a run-time  configurable option.
> > 
> > Frankly, I don't think a configurable  option is a good idea. Druntime
> > has a lot of user configurability, and  as far as I can tell, absolutely
> > none of it has ever been used. It just  adds complexity, both to the code
> > and the documentation.
> 
> I agree.  But also simple does not mean crappy. I think there's too much talk 
>around a  simple matter:
> 
> ======
> Assertion failure should abort the current  unittest block.
> ======
> 
> Do we all agree about the above? We keep on  discussing how the usefulness and 
>informativeness of assertion failures falls  off a cliff after the first 
>failure, and I can't seem to hear loud and clear  that we all want this same 
>thing.

I agree.

> 
> So we've been through these  steps:
> 
> 1. Crappy: failure to assert causes the program to  abort.
> 
> 2. Awful: assert is hacked inside unittests to essentially be  writeln, all 
>unittests run regardless. Programs claims success upon  exit.
> 
> 3. Mediocre: assert continues to be hacked inside unittests to  essentially be 
>writeln + set a flag. Programs exit with errors if the flag was  set.
> 
> NOT YET:
> 
> 4. DESIRED: assert is NOT hacked, any failing assert  ends the current 
>unittest, the failure message is printed, execution continues  with the next 
>unittest, program ends with error code if at least one assert  failed, 
>everybody's happy.

Definitely 4.  Option 1 is workable, but if it takes over a minute to compile 
with unit tests (as dcollections does), then I want to see all unit tests that 
fail so I don't have to go through a tedious 
compile->go-get-a-beer->run-unit-tests->see-if-I'm-sober-enough-to-fix-them 
cycle ;)

Here is one option that I haven't seen yet -- the compiler inserts around all 
unit test blocks the following framework:

try
{
   // run individual unit test block
}
catch(AssertionError ae)
{
   __onAssert(ae);
}

And then we can control exactly what happens on assert.  Someone may create some 
sort of graphical thing, or it could be plugged into an IDE.

-Steve