DMD unittest fail reporting…

Chris Wright via Digitalmars-d digitalmars-d at puremagic.com
Sat Dec 5 09:46:39 PST 2015


On Sat, 05 Dec 2015 11:12:46 +0000, Russel Winder via Digitalmars-d wrote:

> On Sat, 2015-12-05 at 10:24 +0100, Jacob Carlborg via Digitalmars-d
> wrote:
>> […]
>> 
>> You don't want a stack trace for a failed unit test? I have never used
>> a unit test framework that don't output the stack trace for a failed
>> unit test. Why would you want that?
> 
> I put it the other way round: why do you want a stack trace from a
> failure of a unit test?

You seem to be coming from a Go background. D isn't Go.

In Go, a test will fail if you call testing.T.Error[f]. It can crash if 
it runs into a deadlock or some code throws an exception, like on an 
invalid cast or array bounds error or manually calling panic(). When it 
crashes, it gives you a stacktrace (actually, one stacktrace per 
coroutine, and it usually has scads of them active, even if you didn't 
ask for it) and doesn't continue testing anything else.

When I've encountered a panic in Go, the stacktrace was the only thing 
that allowed me to debug the problem instead of throwing my computer out 
the window and then cursing at Rob Pike for a solid hour. The fact that I 
got stacktraces for a dozen unrelated coroutines when I'd never started 
one is pure annoyance, but it's probably helpful for people debugging the 
Go runtime.

In D, you use assert() rather than testing.T.Error[f]. But they aren't 
analogous. D's assert is much closer to Go's panic, except it carries 
with it programmatically readable information on the type of thing that 
caused the panic.

Like panic, assert can happen anywhere.

However, D's assert already gives you one line of stacktrace 
automatically. For simple cases, this is good enough, and the stacktrace 
is just noise.

Since D's AssertError doesn't include the values inside the expression 
that failed, inside any nontrivial case, you need a stacktrace to help 
reconstruct what happened.

Even if it did include that information, let's say you had an invariant 
contract on an object. The invariant contract is called implicitly 
(except in release builds) whenever a public method is called on the 
object. You see that one field on the object is wrong, and you see the 
problematic value. You need a stacktrace in order to have a clue *why* 
the invariant failed.

> The stack trace tells you nothing about the code
> under test that the test doesn't already tell you. All you need to know
> is which tests failed and why.
> 
> Just because some unittests have done something in the past doesn't mean
> it is the right thing to do. The question is what does the programmer
> need for the task at hand. Stack traces add nothing useful to the
> analysis of the test pass or fail.
> 
> I will be looking at dunit, specd and dcheck. The current hypothesis is
> though that the built in unit test is not as good as it needs to be, or
> at least could be.

unittest{}, as Walter has said in the past, isn't intended to have all 
the features you might want. It's intended to be outrageously convenient. 
It's intended to get everyone to write tests when they otherwise wouldn't 
have gone through the trouble.

I seem to recall him advocating for more advanced unittesting libraries, 
even.


More information about the Digitalmars-d mailing list