Extending unittests [proposal] [Proof Of Concept]

[Johannes Pfau]

Current situation:
The compiler combines all unittests of a module into one huge function.
If a unittest in a module fails, the rest won't be executed. The
runtime (which is responsible for calling that per module unittest
method) must always run all unittests of a module.

Goal:
The runtime / test runner can decide for every test if it wants to
continue testing or abort. It should also be possible to run single
tests and skip some tests. As a secondary goal the runtime should
receive the filename and line number of the unittest declaration.

Proposal:
Introduce a new 'MInewunitTest' ModuleInfo flag in object.d and in the
compiler. If MInewunitTest is present, the moduleinfo does not contain
a unittester function. Instead it contains an array (slice) of UnitTest
structs. So the new module property looks like this:
----
@property UnitTest[] unitTest() nothrow pure;
----

the UnitTest struct looks like this:
----
struct UnitTest
{
   string name; //Not used yet
   string fileName;
   uint line;
   void function() testFunc;
}
----

The compiler generates a static array of all UnitTest objects for every
module and sets the UnitTest[] slice in the moduleinfo to point to this
static array. As the compiler already contains individual functions for
every unittest, this isn't too difficult.


Proof of Concept:
I haven't done any dmd hacking before so this might be terrible code,
but it is working as expected and can be used as a guide on how to
implement this:
https://github.com/jpf91/druntime/compare/newUnittest
https://github.com/jpf91/dmd/compare/newUnittest

In this POC the MInewunitTest flag is not present yet, the new method
is always used. Also the implementation in druntime is only minimally
changed. The compiler changes allow an advanced testrunner to do a lot
more:

* Be a GUI tool / use colored output / ...
* Allow to run single, specific tests, skip tests, ...
* Execute tests in a different process, communicate with IPC. This way
  we can deal with segmentation faults in unit tests.

Sample output:
Testing generated/linux/debug/32/unittest/std/array
std/array.d:86          SUCCESS
std/array.d:145         SUCCESS
std/array.d:183         SUCCESS
std/array.d:200         SUCCESS
std/array.d:231         SUCCESS
std/array.d:252         SUCCESS
std/array.d:317         SUCCESS

The perfect solution:
Would allow user defined attributes on tests, so you could name them,
assign categories, etc. But till we have those user defined attributes,
this seems to be a good solution.