RTest, a random testing framework

[Fawzi Mohamed]

On 2008-07-23 06:43:53 +0200, Jesse Phillips <jessekphillips at gmail.com> said:

> [...]
> I agree with Bruce that test cases need to be deterministic. The reason
> for this is that in order to debug on must be able to reproduce the
> problem at hand, if a random is applied that causes an assert to fail,
> you will not be able to track down where the problem lies. Such a system
> is just as bad as running your application and a crash occurring. You
> have successfully produced the random data set needed to create a crash,
> but no way of tracking it down.
> 
> The only way a random test case could be of use is if the random value is
> captured and reported at crash time. This would allow it to be analyzed
> and be added as a static test case to prevent future regressions. I have
> not read the suggested code to see if this is the case, but the adding of
> the test case as an unchanging value is vital to the assurance of bug
> free code.

and this is exactly what my framework does.

It prints the arguments it had generated for the function (often that 
is enough to understand what is wrong) *and* it prints the Rng initial 
state and counter number you need to reproduce exactly that run and (as 
BCS noted, I had written in the initial post) you just need to append a 
.runTests(1,seed,counter) to the test to do it.

The counter number is used to have a full coverage by performing all 
combinations of discrete sets.
For example you know that 0 and 1 will be corner cases for the first 
argument and 2,4,8 for the second argument you can easily define a 
generator that does all possible combinations of them.
You can also mix combinatorial arguments and random ones.

I did this (that for example Quickcheck cannot do easily) because while 
random coverage is good if you have few cases that you want to check, 
the probability that at least one will be missed is greater that what 
one expects, so having both is (I think) a good idea.

Fawzi