RTest, a random testing framework

[JAnderson]

Fawzi Mohamed wrote:
> = RTest
> == RTest a random testing framework
> 
> I wrote a framework to quickly write tests that check property/functions 
> using randomly generated data or all combinations of some values (full 
> coverage).
> This was inspired by Haskell's Quickcheck, but the result is quite 
> different.
> 
> the code is at
>     http://github.com/fawzi/rtest/tree/master
> 
> The idea is to be able to write tests as quickly and as painlessly as 
> possible.
> Typical use is as follow:
> {{{
>    import frm.rtest.RTest;
> 
>    private mixin testInit!() autoInitTst;
> 
>    void myTests(){
>        // define a collection for my tests
>        TestCollection myTests=new 
> TestCollection("myTests",__LINE__,__FILE__);
> 
>        // define a test
>        autoInitTst.testTrue("testName",functionToTest,__LINE__,__FILE__);
>        // for example
>        autoInitTst.testTrue("(2*x)%2==0",(int x){ return 
> ((2*x)%2==0);},__LINE__,__FILE__);
> 
>        // run the tests
>        myTests.runTests();
>    }
> }}}
> If everything goes well not much should happen, because by default the 
> printer does not write successes.
> You can change the default controller as follows:
> {{{
>    SingleRTest.defaultTestController=new TextController(
>        TextController.OnFailure.StopTest,
>        TextController.PrintLevel.AllShort,Stdout);
> }}}
> and it should write out something like
> {{{
>    test`testName`          
> failures-passes/totalTests(totalCombinatorialRuns)
> }}}
> i.e.:
> {{{
>    test`assert(x*x<100)`                 0-100/100(100)
>    test`assert(x*x<100)`                 0- 56/100(100)
> }}}
> If one wants to run three times as many tests:
> {{{
>    myTests.runTests(3);
> }}}
> If a test fails then it will print out something like this
> {{{
>    test`(2*x)%4==0 || (2*x)%4==2` failed (returned false instead of true)
>    arg0: -802454419
> 
>    To reproduce:
>     intial rng state: 
> CMWC000000003ade6df6_00000020_595a6207_2a7a7b53_e59a5471_492be655_75b9b464_f45bb6b8_c5af6b1d_1eb47eb9_ff49627d_fe4cecb1_fa196181_ab208cf5_cc398818_d75acbbc_92212c68_ceaff756_c47bf07b_c11af291_c1b66dc4_ac48aabe_462ec397_21bf4b7a_803338ab_c214db41_dc162ebe_41a762a8_7b914689_ba74dba0_d0e7fa35_7fb2df5a_3beb71fb_6dcee941_0000001f_2a9f30df_00000000_00000000 
> 
> 
> 
>    counter: [0]
>    ERROR test `(2*x)%4==0 || (2*x)%4==2` from `test.d:35` FAILED!!
>    -----------------------------------------------------------
>    test`(2*x)%4==0 || (2*x)%4==2`   1-  0/  1(  1)
> }}}
> from it you should see the arguments that made the test fail.
> If you want to re-run it you can add .runTests(1,seed,counter) to it, i.e.:
> {{{
> autoInitTst.testTrue("(2*x)%4==0 || (2*x)%4==2 (should fail)",(int x){ 
> return ((2*x)%4==0 || (2*x)%4==2);},
>    
> __LINE__,__FILE__).runTests(1,"CMWC000000003ade6df6_00000020_595a6207_2a7a7b53_e59a5471_492be655_75b9b464_f45bb6b8_c5af6b1d_1eb47eb9_ff49627d_fe4cecb1_fa196181_ab208cf5_cc398818_d75acbbc_92212c68_ceaff756_c47bf07b_c11af291_c1b66dc4_ac48aabe_462ec397_21bf4b7a_803338ab_c214db41_dc162ebe_41a762a8_7b914689_ba74dba0_d0e7fa35_7fb2df5a_3beb71fb_6dcee941_0000001f_2a9f30df_00000000_00000000",[0]) 
> 
> }}}
> 
> If
> the default generator is not good enough you can create tests that use a 
> custom generator like this:
> {{{
>    private mixin testInit!(manualInit,checkInit) customTst;
> }}}
> in manualInit you have the following variables:
>  arg0,arg1,... : variable of the first,second,... argument that you can 
> initialize
>  arg0_i,arg0_i,... : index variable for combinatorial (extensive) coverage.
>    if you use it you probably want to initialize the next variable
>  arg0_max, arg1_max,...: variable that can be initialized to an integer 
> that gives
>    the maximum value of arg0_i+1, arg1_i+1,... giving it a value makes 
> the combinatorial
>    machine work, and does not set test.hasRandom to true for this variable
> If an argument is not defined the default generation procedure
> {{{
>    Rand r=...;
>    argI=generateRandom!(typeof(argI))(r);
> }}}
> is used.
> checkInit can be used if the generation of the random configurations is 
> mostly good,
>  but might contain some configurations that should be skipped. In 
> checkInit one
>  should set the boolean variable "acceptable" to false if the configuration
>  should be skipped.
> 
> For example:
> {{{
>    private mixin testInit!("arg0=r.uniformR(10);") smallIntTst;
> }}}
> then gets used as follow:
> {{{
>    smallIntTst.testTrue("x*x<100",(int x){ return 
> (x*x<100);},__LINE__,__FILE__).runTests();
> }}}
> by the way this is also a faster way to perform a test, as you can see 
> you don't need to define a collection (but probably it is a good idea to 
> define one)
> 
> enjoy
> 
> Fawzi Mohamed
> 

Nice! although I'm not exactly sure what the process is above with your 
code.  I've often though about writing a tool that automatically creates 
unit-tests.  Something that you could give an object and it would test 
every function in many random ways and sequences.  It would validate 
based on the outputs of the function.

So more or less it would be a test to make sure that the function 
behavior doesn't change, rather then a check to make sure it works in 
the first place. However you could look at the results that where 
generated and touch it up a little to give it better coverage.

-Joel