DMD unittest fail reporting…
Chris Wright via Digitalmars-d
digitalmars-d at puremagic.com
Sat Dec 5 09:46:39 PST 2015
On Sat, 05 Dec 2015 11:12:46 +0000, Russel Winder via Digitalmars-d wrote:
> On Sat, 2015-12-05 at 10:24 +0100, Jacob Carlborg via Digitalmars-d
> wrote:
>> […]
>>
>> You don't want a stack trace for a failed unit test? I have never used
>> a unit test framework that don't output the stack trace for a failed
>> unit test. Why would you want that?
>
> I put it the other way round: why do you want a stack trace from a
> failure of a unit test?
You seem to be coming from a Go background. D isn't Go.
In Go, a test will fail if you call testing.T.Error[f]. It can crash if
it runs into a deadlock or some code throws an exception, like on an
invalid cast or array bounds error or manually calling panic(). When it
crashes, it gives you a stacktrace (actually, one stacktrace per
coroutine, and it usually has scads of them active, even if you didn't
ask for it) and doesn't continue testing anything else.
When I've encountered a panic in Go, the stacktrace was the only thing
that allowed me to debug the problem instead of throwing my computer out
the window and then cursing at Rob Pike for a solid hour. The fact that I
got stacktraces for a dozen unrelated coroutines when I'd never started
one is pure annoyance, but it's probably helpful for people debugging the
Go runtime.
In D, you use assert() rather than testing.T.Error[f]. But they aren't
analogous. D's assert is much closer to Go's panic, except it carries
with it programmatically readable information on the type of thing that
caused the panic.
Like panic, assert can happen anywhere.
However, D's assert already gives you one line of stacktrace
automatically. For simple cases, this is good enough, and the stacktrace
is just noise.
Since D's AssertError doesn't include the values inside the expression
that failed, inside any nontrivial case, you need a stacktrace to help
reconstruct what happened.
Even if it did include that information, let's say you had an invariant
contract on an object. The invariant contract is called implicitly
(except in release builds) whenever a public method is called on the
object. You see that one field on the object is wrong, and you see the
problematic value. You need a stacktrace in order to have a clue *why*
the invariant failed.
> The stack trace tells you nothing about the code
> under test that the test doesn't already tell you. All you need to know
> is which tests failed and why.
>
> Just because some unittests have done something in the past doesn't mean
> it is the right thing to do. The question is what does the programmer
> need for the task at hand. Stack traces add nothing useful to the
> analysis of the test pass or fail.
>
> I will be looking at dunit, specd and dcheck. The current hypothesis is
> though that the built in unit test is not as good as it needs to be, or
> at least could be.
unittest{}, as Walter has said in the past, isn't intended to have all
the features you might want. It's intended to be outrageously convenient.
It's intended to get everyone to write tests when they otherwise wouldn't
have gone through the trouble.
I seem to recall him advocating for more advanced unittesting libraries,
even.
More information about the Digitalmars-d
mailing list