RTest, a random testing framework

[JAnderson]

Fawzi Mohamed wrote:
> On 2008-07-23 06:43:53 +0200, Jesse Phillips <jessekphillips at gmail.com> 
> said:
> 
>> [...]
>> I agree with Bruce that test cases need to be deterministic. The reason
>> for this is that in order to debug on must be able to reproduce the
>> problem at hand, if a random is applied that causes an assert to fail,
>> you will not be able to track down where the problem lies. Such a system
>> is just as bad as running your application and a crash occurring. You
>> have successfully produced the random data set needed to create a crash,
>> but no way of tracking it down.
>>
>> The only way a random test case could be of use is if the random value is
>> captured and reported at crash time. This would allow it to be analyzed
>> and be added as a static test case to prevent future regressions. I have
>> not read the suggested code to see if this is the case, but the adding of
>> the test case as an unchanging value is vital to the assurance of bug
>> free code.
> 
> and this is exactly what my framework does.
> 
> It prints the arguments it had generated for the function (often that is 
> enough to understand what is wrong) *and* it prints the Rng initial 
> state and counter number you need to reproduce exactly that run and (as 
> BCS noted, I had written in the initial post) you just need to append a 
> .runTests(1,seed,counter) to the test to do it.
> 
> The counter number is used to have a full coverage by performing all 
> combinations of discrete sets.
> For example you know that 0 and 1 will be corner cases for the first 
> argument and 2,4,8 for the second argument you can easily define a 
> generator that does all possible combinations of them.
> You can also mix combinatorial arguments and random ones.
> 
> I did this (that for example Quickcheck cannot do easily) because while 
> random coverage is good if you have few cases that you want to check, 
> the probability that at least one will be missed is greater that what 
> one expects, so having both is (I think) a good idea.
> 
> Fawzi
> 


At work I use UnitTest++.  That allows me to run the program though a 
debugger when I need to.  Often its simply enough to know that you 
changed something that broke the unittest however perhaps you could 
print out the entire unit test when it fails.  Then people could simply 
run that piece of code though a debugger.

Another though that I've thought would be cool for these sort of unit 
tests is something that monitors/records your code and automatically 
generates units tests on the macro scale.  ie it would write unit test 
files out which would essentially be recording of what your app did.

I imagine you could come up with some sort of template to do this:

ie:

I have:

foo(5);
foo2(20);

Now I want to record foo and foo2 as well so I change the definition to:

Record(foo)(5);  //Run foo and record it
Record(foo2)(10);//Run foo2 and record it

//Output from application run
foo(5);
foo2(10);

Of course this would also be useful for playing back code in smoke 
tests.  It could of course be made more advanced, like being able to 
change the script to wait for a certain function to return a certain 
result etc...

-Joel