Improving unit tests

[Christopher Wright]

Janderson wrote:
> Questions:
> 1) What features are missing from D's unit tests that you miss?

- Named tests
- Error reporting
- The ability to list unit tests
- The ability to get different types of output
- Continuous integration support
- Running a subset of the tests
- Not running main after running tests
- Not running tests before main, and not having to recompile to get this 
behavior
- A setup/teardown structure with common code to be run before and after 
a particular set of tests. This one won't happen without a major 
redesign of the builtin unittests, though.

Of those, Dunit has named tests and good error reporting. It has 
continuous integration support. It can list tests without running them. 
I'm adding support for running particular subsets with the default test 
runner right now (test runners can do this, but there's no command line 
argument to support it). And of course it has setup/teardown methods.

Dunit replaces main, so main doesn't run after it, and it doesn't run 
before main.

> 2) Should D decouple unittests from the language or should there be 
> language changes to allow for for more extensible unit tests?

D should decouple unittests from the runtime.

> 3) If you chose "more extensible unit tests" what language features 
> would be needed to make this happen.

1. Named unittests.
There are a few reasons for this:
	- Your code should be self-documenting. unittest {} tells you nothing.
	- If a test fails, you want to know which test it is. A backtrace 
library helps more than a unittest name, I admit, but both are useful.
	- Naming for unittests integrates better with continuous integration 
servers like CruiseControl and CCNET -- these expect unittests to have 
fully qualified names.

2. A pluggable unittest runner.
With this, any functionality that unittests lack can be provided. The 
unittest runner has to be in the runtime, of course.

3. Each test should be its own function.
Currently, all unittest blocks in any module are conglomerated into one 
function, ModuleInfo.unitTest. ModuleInfo should have a list of 
name/function pairs. This is required for named unittests, and for 
reporting errors on a per-test basis.

4. The unittest runner must be able to have runtime arguments.
At work, we use test-driven development. We have about 5700 classes, 
half of which are test classes. That's 7,200 tests currently. They take 
ten minutes to run. You can't do TDD with ten minute lag for running tests.

Command line arguments are the easiest way of changing options at 
runtime. The alternatives are some sort of interactive session with 
stdio, an options file, or recompiling (which isn't runtime). None of 
these is particularly good.