RTest, a random testing framework

[Fawzi Mohamed]

On 2008-07-22 00:12:51 +0200, BCS <ao at pathlink.com> said:

> Reply to Fawzi,
> 
> I'm not quite following that, it seems to randomly select test values 
> to supply to a function.

Hi BCS,

that is exactly the idea: test a function with random values as input, 
so that many test cases can be generated with little effort from your 
side (I dislike very much writing test cases :).
These random values do not have to be basic types, they can be 
structures, classes,typedefs, actually if one is using too much custom 
generators probably he should define his type, and write
T randomGenerate(T:MyType) for it.

> While I like that Idea, the implementation doesn't appeal to me (I have 
> never liked string mixins if anything else is usable).

I agree the string mixins are a feature that should be avoided if 
possible, but I didn't see how to do it with a similar effort without 
them.
I see them as the equivalent of dynamically typed languages vs 
statically typed, like in that case you loose many checks, but you can 
gain simplicity and expressiveness.
There are many discussions about it but it seems clear to me that each 
choice has it own pros and cons, and depending on the problem it might 
be more or less suited.

>  I had an idea a while ago to might make for a better interface and 
> might even allow better application of constraints:
> 
> double x,y;
> Assert(1/x=pow(x,-1)).
>     Where(x).NotZero.
>     TestRandom();
> 
> Assert(x<y).
>     Where(x).LessThan(y).
>     Where(x).InRange(-10,10).
>     TestEdges(1000); //< 1000 test points
> 
> Assert's arg would be lazy bool and the delegate would be storeed, the 
> Where would use ref so it can pick up a pointer to x. Some sort of 
> internal magic would keep track of what constraints apply to what and 
> let the Test* functions inelegantly search the envelope.

I like your proposal but I fail to see how it can scale to more complex cases.

Indeed my examples where very simple, but a "real" case looks like this:
You have a function that solves linear systems of equations, and you 
want to test it.
The matrix should be square, and the b vector should have the same size 
as the matrix dimension.
So either you define  an ad-hoc structure, or you write a custom 
generator for it
(it is quite unlikely that the constraint are satisfied just by chance 
and you would spend all your time waiting for a valid test case).
Then (if detA>0) you can check that the solution really solves the 
system of equations with a small residual error.
Your test can fail in many ways also due to the internal checks of the 
equation solver, and you want to  always have a nice report that lets 
you reproduce the problem.
Another typical use case is when you have a slow reference 
implementation for something and a fast one, and you want to be sure 
they are the same.

I think that my approach works well in those cases, and I don't see how 
your "magic" could work, but I would like to be shown wrong :)

Fawzi