[phobos] Silent failure of std.container unittests

[Sean Kelly]

On Jul 15, 2010, at 6:57 AM, Steve Schveighoffer wrote:
> 
> ----- Original Message ----
>> From: Andrei Alexandrescu <andrei at erdani.com>
>> To: Discuss the phobos library for D <phobos at puremagic.com>
>> Sent: Wed, July 14, 2010 10:42:06 PM
>> Subject: Re: [phobos] Silent failure of std.container unittests
>> 
>> On 07/14/2010 08:58 PM, Walter Bright wrote:
>>> 
>>> 
>>> Sean  Kelly wrote:
>>>> 
>>>> 
>>>> How about this... since unit  tests are run after static ctors are run,
>>>> the behavior of whether to  throw an AssertError or report and continue
>>>> can be a run-time  configurable option.
>>> 
>>> Frankly, I don't think a configurable  option is a good idea. Druntime
>>> has a lot of user configurability, and  as far as I can tell, absolutely
>>> none of it has ever been used. It just  adds complexity, both to the code
>>> and the documentation.
>> 
>> I agree.  But also simple does not mean crappy. I think there's too much talk 
>> around a  simple matter:
>> 
>> ======
>> Assertion failure should abort the current  unittest block.
>> ======
>> 
>> Do we all agree about the above? We keep on  discussing how the usefulness and 
>> informativeness of assertion failures falls  off a cliff after the first 
>> failure, and I can't seem to hear loud and clear  that we all want this same 
>> thing.
> 
> I agree.
> 
>> 
>> So we've been through these  steps:
>> 
>> 1. Crappy: failure to assert causes the program to  abort.
>> 
>> 2. Awful: assert is hacked inside unittests to essentially be  writeln, all 
>> unittests run regardless. Programs claims success upon  exit.
>> 
>> 3. Mediocre: assert continues to be hacked inside unittests to  essentially be 
>> writeln + set a flag. Programs exit with errors if the flag was  set.
>> 
>> NOT YET:
>> 
>> 4. DESIRED: assert is NOT hacked, any failing assert  ends the current 
>> unittest, the failure message is printed, execution continues  with the next 
>> unittest, program ends with error code if at least one assert  failed, 
>> everybody's happy.
> 
> Definitely 4.  Option 1 is workable, but if it takes over a minute to compile 
> with unit tests (as dcollections does), then I want to see all unit tests that 
> fail so I don't have to go through a tedious 
> compile->go-get-a-beer->run-unit-tests->see-if-I'm-sober-enough-to-fix-them 
> cycle ;)
> 
> Here is one option that I haven't seen yet -- the compiler inserts around all 
> unit test blocks the following framework:
> 
> try
> {
>   // run individual unit test block
> }
> catch(AssertionError ae)
> {
>   __onAssert(ae);
> }
> 
> And then we can control exactly what happens on assert.  Someone may create some 
> sort of graphical thing, or it could be plugged into an IDE.

Huh, this is kind of like the overridable assert handler in druntime already, except that it's actually useful :-)  For now, this could be achieved by overriding the module unit tester however.  It's just a bit bulkier than simply overriding the reporting mechanism.