Summary on unit testing situation

[Fawzi Mohamed]

On 23-mar-10, at 20:29, bearophile wrote:

> Pelle M.:
>
>> I'm not sure I understand, could you explain?<
>
> That was my best explanation, sorry.
>
>
>> I am not experienced with unittest frameworks, and would like to  
>> understand what the D system lacks.<
>
> I think two times in the past I have written a list of those lacking  
> things. To give a good answer to your question I have to write a  
> lot, and it's not nice to write a lot when the words get ignored. So  
> first devs have to agree that a problem exists, then later we can  
> design things to improve the situation. Otherwise it's just a waste  
> of my energy, like trying to talk in vacuum.

actually there are some hooks in tango (and I believe similarly in  
phobos) to do what you want

the module info contains the unittests and you can replace the default  
for example the unittester of tango looks like this

import all modules to be tested
import tango.io.Stdout;
import tango.core.Runtime;
import tango.core.stacktrace.TraceExceptions;

bool tangoUnitTester()
{
     uint countFailed = 0;
     uint countTotal = 1;
     Stdout ("NOTE: This is still fairly rudimentary, and will only  
report the").newline;
     Stdout ("    first error per module.").newline;
     foreach ( m; ModuleInfo )  // _moduleinfo_array )
     {
         if ( m.unitTest) {
             Stdout.format ("{}. Executing unittests in '{}' ",  
countTotal, m.name).flush;
             countTotal++;
             try {
                m.unitTest();
             }
             catch (Exception e) {
                 countFailed++;
                 Stdout(" - Unittest failed.").newline;
                 e.writeOut(delegate void(char[]s){ Stdout(s); });
                 continue;
             }
             Stdout(" - Success.").newline;
         }
     }

     Stdout.format ("{} out of {} tests failed.", countFailed,  
countTotal - 1).newline;
     return true;
}

static this() {
     Runtime.moduleUnitTester( &tangoUnitTester );
}

void main() {}

one can do something fancier if he wants.
To really have all test one would need to have an array (or iterator)  
in the module information instead of a single global unittest  
function. Alternatively one could pass some flags to the unittest  
function to control its execution.

> Unit testing has to continue when tests fail. All code must be  
> testable, compile-time code too. You need a way to assert that  
> things go wrong too, like exceptions, asserts, compile-time asserts,  
> etc when they are designed to. It's good to have a way to give a  
> name to tests. And unit test systems enjoy some reflection to  
> organize themselves, to attach tests to code automatically. During  
> development you want to test only parts of the code, not the whole  
> program. Unit testing OOP code has other needs, because in a test  
> you may need to break data hiding of classes and structs. If you  
> unit test hundred of classes you soon find the necessity of  
> something to help creation of fake testing objects. You need some  
> tools for creating mock test objects (objects that simulate external  
> resources). You need a help to perform performance tests, to print  
> reports of the testing. You need layers of testing, slow tests and  
> quick tests that you can run every few minutes or seconds of  
> programming.!
>  Generally the more the unit test system does automatically the  
> better it is, because you want to write and use unit tests in the  
> most fast way possible. Those things are useful, but putting most of  
> those things inside a compiler is not a good idea.

I think that what you want is beyond normal requests, executing all  
tests, tests of one module, a single test, yes that should be  
relatively simple.
More complex test series/combination are probably better served by a  
specialized regression tester.

Actually I use a specialized tester that is parallel, and whose basic  
testing building block is a testing function in which the arguments  
for it are generated automatically (derived types have to implement  
generating functions).
This is a somewhat different way to look at tests that the usual one  
(inspired from haskell's QuickCheck), but one that I prefer.
In the end the power is the same, instead of fixtures to prepare a  
test environment you can define a derived type whose generating  
function do the fixtures, and then have the tests as function having  
that type as argument.

Test suites I normally organize like the package structure.

What I have is something like that would also be useful in tango/ 
phobos is a pre written main like function, so that one can easily  
create test suites for pieces of code.
I have for example
	int mainTestFun(char[][] argStr,SingleRTest testSuite)
which can be used to create a unittester that recognizes flags to  
initialize it, perform subtests,...

Fawzi