DMD unittest fail reporting…

Russel Winder via Digitalmars-d digitalmars-d at puremagic.com
Sat Dec 5 12:44:53 PST 2015


On Sat, 2015-12-05 at 17:46 +0000, Chris Wright via Digitalmars-d
wrote:
> 
[…]
> You seem to be coming from a Go background. D isn't Go.

I'm from a FORTRAN IV, FORTRAN G, Algol 68, Pascal, C, FORTRAN 77, C++,
Miranda, Lisp, Scheme, Modula-2, Java, Python, Scala, Groovy, D,
Haskell, Ceylon, Kotlin, Clojure, Go, Frege background, but to mention
this would just seem to be bragging. ;-)
 
> In Go, a test will fail if you call testing.T.Error[f]. It can crash
> if 
> it runs into a deadlock or some code throws an exception, like on an 
> invalid cast or array bounds error or manually calling panic(). When
> it 
> crashes, it gives you a stacktrace (actually, one stacktrace per 
> coroutine, and it usually has scads of them active, even if you
> didn't 
> ask for it) and doesn't continue testing anything else.

For the purposes of this argument, let's ignore crashes or manually
executed panics. The issue is the difference in behaviour between
assert and Errorf. assert in languages that use it causes an exception
and this causes termination which means execution of other tests does
not happen unless the framework makes sure this happens. D unittest
does not. Errorf notes the failure and carries on, this is crucial
important for good testing using loops. In none of these case should a
stack trace ever be reported since it is irrelevant to the testing
scenario.

> When I've encountered a panic in Go, the stacktrace was the only
> thing 
> that allowed me to debug the problem instead of throwing my computer
> out 
> the window and then cursing at Rob Pike for a solid hour. The fact
> that I 
> got stacktraces for a dozen unrelated coroutines when I'd never
> started 
> one is pure annoyance, but it's probably helpful for people debugging
> the 
> Go runtime.

To be honest I am not sure how this rant contributes to the discussion
of testing.

> In D, you use assert() rather than testing.T.Error[f]. But they
> aren't 
> analogous. D's assert is much closer to Go's panic, except it carries
> with it programmatically readable information on the type of thing
> that 
> caused the panic.

Very true, and that is core to the issue here. asserts raise exceptions
which , unless handled by the testing framework properly, cause
termination. This is at the heart of the problem. For data-driven
testing some form of loop is required. The loop must not terminate if
all the tests are to run. pytest.mark.parametrize does the right thing,
as do normal loops and Errorf. D assert does the wrong thing.

> Like panic, assert can happen anywhere.

Tecnically, I guess yes, but…

> However, D's assert already gives you one line of stacktrace 
> automatically. For simple cases, this is good enough, and the
> stacktrace 
> is just noise.

But it stops the loop. This makes the loop a fundamentally useless
construct in tests. This is at the heart of the problem with the
unittest construct.

> Since D's AssertError doesn't include the values inside the
> expression 
> that failed, inside any nontrivial case, you need a stacktrace to
> help 
> reconstruct what happened.

I think this is the evidence that proves that the current D testing
framework is in need of work to make it better than it is currently.

> Even if it did include that information, let's say you had an
> invariant 
> contract on an object. The invariant contract is called implicitly 
> (except in release builds) whenever a public method is called on the 
> object. You see that one field on the object is wrong, and you see
> the 
> problematic value. You need a stacktrace in order to have a clue
> *why* 
> the invariant failed.

If a stacktrace is needed the testing framework is inadequate.

[…]
> 
> unittest{}, as Walter has said in the past, isn't intended to have
> all 
> the features you might want. It's intended to be outrageously
> convenient. 
> It's intended to get everyone to write tests when they otherwise
> wouldn't 
> have gone through the trouble.

Being outrageously convenient means it needs to be the tool of choice.
Which currently it isn't really for anything other than trivial
example-based testing on a case-by-case basis.

> I seem to recall him advocating for more advanced unittesting
> libraries, 
> even.

dunit (being a Delphi framework) doesn't fit the bill :-)

dunit being a xUnit clone doesn't really work in the same way that
cppUnit doesn't. Catch is the current C++ front-runner testing
framework, there should be a D equivalent. dspecs and specd are
interesting but I am not sure they are being developed. dcheck is great
stuff and it is sad that it has been dormant for 2 years according to
the GitHub repository record.

In writing this I realize I have supped once too often of the Ricard. I
shall now switch to red wine and cease emailing.
 
-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20151205/bb01adbe/attachment.sig>


More information about the Digitalmars-d mailing list