RTest, a random testing framework

[BCS]

Reply to Fawzi,

> On 2008-07-22 00:12:51 +0200, BCS <ao at pathlink.com> said:
> 
>> Reply to Fawzi,
>> 
>> I'm not quite following that, it seems to randomly select test values
>> to supply to a function.
>> 
> Hi BCS,
> 
> that is exactly the idea: test a function with random values as input,
> so that many test cases can be generated with little effort from your
> side (I dislike very much writing test cases :).
> These random values do not have to be basic types, they can be
> structures, classes,typedefs, actually if one is using too much custom
> generators probably he should define his type, and write
> T randomGenerate(T:MyType) for it.
>> While I like that Idea, the implementation doesn't appeal to me (I
>> have never liked string mixins if anything else is usable).
>> 
> I agree the string mixins are a feature that should be avoided if
> possible, but I didn't see how to do it with a similar effort without
> them.
> I see them as the equivalent of dynamically typed languages vs
> statically typed, like in that case you loose many checks, but you can
> gain simplicity and expressiveness.
> There are many discussions about it but it seems clear to me that each
> choice has it own pros and cons, and depending on the problem it might
> be more or less suited.
>> I had an idea a while ago to might make for a better interface and
>> might even allow better application of constraints:
>> 
>> double x,y;
>> Assert(1/x=pow(x,-1)).
>> Where(x).NotZero.
>> TestRandom();
>> Assert(x<y).
>> Where(x).LessThan(y).
>> Where(x).InRange(-10,10).
>> TestEdges(1000); //< 1000 test points
>> Assert's arg would be lazy bool and the delegate would be storeed,
>> the Where would use ref so it can pick up a pointer to x. Some sort
>> of internal magic would keep track of what constraints apply to what
>> and let the Test* functions inelegantly search the envelope.
>> 
> I like your proposal but I fail to see how it can scale to more
> complex cases.
> 
> Indeed my examples where very simple, but a "real" case looks like
> this:
> You have a function that solves linear systems of equations, and you
> want to test it.
> The matrix should be square, and the b vector should have the same
> size
> as the matrix dimension.
> So either you define  an ad-hoc structure, or you write a custom
> generator for it
> (it is quite unlikely that the constraint are satisfied just by chance
> and you would spend all your time waiting for a valid test case).
> Then (if detA>0) you can check that the solution really solves the
> system of equations with a small residual error.
> Your test can fail in many ways also due to the internal checks of the
> equation solver, and you want to  always have a nice report that lets
> you reproduce the problem.
> Another typical use case is when you have a slow reference
> implementation for something and a fast one, and you want to be sure
> they are the same.
> I think that my approach works well in those cases, and I don't see
> how your "magic" could work, but I would like to be shown wrong :)
> 

The way the magic above would work is that the Where clauses are not just 
tested, they are used to define the test envelope. e.g. the clause Where(x).InRange(-10,10) 
would define a radome number generator that sets x to a radome number in 
the given range, then the Where(x).LessThan(y) clause would be evaluated 
and a radome number generated for y that is always in the specified range 
(in this cases [-inf, x]). Thus in the given cases, ALL test cases would 
be valid.

Other 'Where' predicates could be defended that don't drive the generator 
but are only checked to see if the test cases is valid (like the det()!=0 
example you gave) or even only used to determine the expected result so that 
correct failure cases could be checked as well.

Other variants of the where clause could be used like:

 With(arg).InRange(0,8).Eval(matrix.SquareArrray(arg)). // set arg in [0,10] 
then eval the expression

The overloading needed to mix ref and lazy parameters with constant arguments 
would take some work but I think it is doable. There would also need to be 
quite a bit of magic in the final test function to find test cases within 
the provided constraints but for many cases this could be made a lot better 
than guess and check.

I would love to take a crack at the problem but I have about 4 different 
project in the queue already.

> Fawzi
> 

p.s. typo: Assert(1/x=pow(x,-1)).  -> Assert(1/x==pow(x,-1)).