DMD unittest fail reporting…

[Chris Wright via Digitalmars-d]

On Sat, 05 Dec 2015 20:44:53 +0000, Russel Winder via Digitalmars-d wrote:

> If a stacktrace is needed the testing framework is inadequate.

But there are problems with saying that the builtin assert function 
should show the entire expression with operand values, nicely formatted.

assert has to serve both unittesting and contract programming. When 
dealing with contract programming and failed contracts, you risk objects 
being in invalid states. Trying to call methods on such objects in order 
to provide descriptive error messages is risky. A helpful stacktrace 
might be transformed into a segmentation fault, for instance. Or an 
assert error might be raised while attempting to report an assert error.

assert is a builtin function. It's part of the runtime. That puts rather 
strict constraints on how much it can do. The runtime can't depend on the 
standard library, for instance, so if you want assert() to include the 
values that were problematic, the runtime has to include that formatting 
code. That doesn't seem like a lot on its own, but std.format is probably 
a couple thousand lines of code. (About 3,000 semicolons, including 
unittests.)

I would like these nicely formatted messages. I don't think it's 
reasonably practical to add them to assert. I'll spend some thought on 
how to implement them outside the runtime, for a testing framework, 
though I'm not optimistic on a nice API. Catch does it with macros and by 
parsing C++, and the nearest equivalent in D is string mixins, which are 
syntactically more complex. Spock does it with a compiler plugin. I know 
I can do it with strings and string mixins, but that's not exactly going 
to be a clean API.

>> unittest{}, as Walter has said in the past, isn't intended to have all
>> the features you might want. It's intended to be outrageously
>> convenient.
>> It's intended to get everyone to write tests when they otherwise
>> wouldn't have gone through the trouble.
> 
> Being outrageously convenient means it needs to be the tool of choice.

I don't see why those would need to be related. To be useful, it has to 
be good enough to catch errors and convenient enough to induce people to 
test when they wouldn't otherwise.

> Which currently it isn't really for anything other than trivial
> example-based testing on a case-by-case basis.

Most unittesting is a developer testing a handful of specific cases that 
seem interesting or useful to test. It's easy, and it's usually good 
enough.

Property testing like in dcheck requires you to be able to produce a 
series of algorithms that jointly create input/output pairs that cover 
the entire domain of the function under test. That's often difficult. And 
it's mainly applicable for pure functions, not interaction-based testing.

Random example: I wrote a URL parsing library. How would I test it using 
dcheck? I would come up with a series of URL parts according to patterns 
that I thought up and test based on those. That's basically identical to 
coming up with specific URLs myself, except I have to implement a second 
algorithm to join parts into a URL. The coverage is pretty much the same. 
I get to see if it barfs on some strange character, or more likely I'll 
find that my test URL generator produces invalid URLs.

That would cost me a lot more time and thought than just writing out a 
few examples. It would have coverage holes because of valid URL patterns 
I hadn't thought of. Overall no benefit.

On the other hand, as part of this library I implemented a punycode codec. 
I can use dcheck or a similar system to generate random valid Unicode 
strings. I can't use this to ensure that my codec is correct relative to 
the RFC, but I can at least ensure that the encoding is self-consistent.

So it's a good tool to have available, but I do need other ways to test 
my code.