Unit tests in D

[Michel Fortin]

On 2010-05-05 14:24:37 -0400, bearophile <bearophileHUGS at lycos.com> said:

> My post about unit test features contains other things worth discussing 
> about. This was just the first point :-)

Indeed.

I agree it'd be handy to have named unit tests. I see two reasons to 
use named unit tests:

1. to print a list of all the tests as they run
2. to print a list of the tests that fails

Currently the output of a unittest is to print the first assertion that 
fails. I think that's a good default, but if you add a name to your 
unit test it can give you more context when it fails. Here's a way to 
do it with the current syntax:

	unittest {
		scope (failure) writeln("Calculate pi using method X: FAIL");
	
		auto result = methodX();
		assert(result == 3.1416);
	}

If the test fails, you'll get this output:

	Calculate pi using method X: FAIL
	file.d(5): assertion failure pi == 3.1416

which is a noticeable improvement because you don't have to go and look 
at the file to know what this test is about.

If the test pass, it'll will output nothing. Whether we want to output 
a line for every test, I'm not sure. D encourages small unit tests 
scattered all around the place, and I'd be worried that being too 
verbose when the tests are run would discourage people from running the 
tests in the first place. On the other hand, it's useful, especially 
when a test hangs, takes too long, or crashes the program, to be able 
to see the list of all the tests as they run.

So what I would suggest is two things. A way to attach a name to a unit 
test, like this:

	unittest "Calculate pi using method x" {
		...
	}

This is better than scope (failure) because the test runner is now in 
charge of deciding what to do with each test. I'd suggest that the 
runtime print the name of a test when it fails:

	Calculate pi using method X: FAIL
	file.d(5): assertion failure pi == 3.1416

If the environment variable D_VERBOSE_UNITTEST is set when the program 
is run, the runtime could print the name of each test as it executes 
the test, followed by "PASS" upon successful completion or "FAIL" on 
failure:

	Calculate e using method X: PASS
	Calculate pi using method X: FAIL
	file.d(5): assertion failure pi == 3.1416

The environment variable makes sure that no one is bothered by a long 
list of tests unless they explicitly ask for it. When you want to see 
which tests are run and get a general feel of the time they take.

Any other use for named unit tests?

-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/