'live' testing style

[spir]

Hello,


Back to the subject of using unittest and assert. I'll try to illustrate a 
testing style that seems to be rare in the D community, is not properly 
supported by D builtin tools --while this would require only minimal 
improvements--, and is imo rather sensible, practicle and efficient. Using an 
example of a current project of mine (a toy homoiconic language, kind of 
special Lisp).

Below is a copy of part of the parser's test suite. Maybe have a quick look 
before reading further [Note: I don't mean the example is a model of any sort, 
not even it's good code. It is just illustration of some testing practice.] The 
testList func tests the match func for List literal notations. Some comments, 
in form of question-answer:

* Why isn't testList a unittest block?

Using named funcs, I can switch on & off specific test suites by (un)commenting 
their call from the main and unique unittest block. Else, either they all run, 
or none. During development, I only keep active the test func(s) relative to 
the feature I'm currently working on.
Remedy: named unittests.

* Why those write(f)(ln) in unitests?

For me unittests are not only regression tests, to be used /after/ development 
for maintenance purpose. Instead, they are the #1 and necessary source of 
information about what the code actually does, how, and why. I constantly run 
tests, for feedback, just to check all runs fine, or diagnose what goes wrong.
Thus, the first method I write for a type if often toString; I give it a form 
which tells me all I need about an object; commonly, it's just the notation to 
re-create it. This, and having test in mind for writing toString, also helps in 
properly defining the type itself.
For diagnosis, this information is of primordial value. And, against common 
sense maybe, data about what goes right, or seems to, is commonly even more 
important than data about what goes wrong, to determine the cause(s). (*)
Remedy: verbose asserts on request.

* What are those '// for diagnose' statement?

Since the test func has a form of loop over a test suite, a failing assert 
would fail (!) to tell which case is in cause, instead miserably spitting out a 
line number only. To get the proper info, I just need to comment out those few 
lines of code, which would output for instance (artificial failure case):
...
`[nil false  true]` --> [nil false true]
`["a"  "ab"   "abc"]` --> ["a" "ab" "abc"]
`[[3] [3 2] [3 1 2]]` --> [[3] [3 2] [3 1 2]]
*** error *** expected: [[3] [32] [3 1 2]]
Remedy: improved assertion failure message form.

* Why string'ed results?

In this case, like in many others due to various reasons, it's uneasy to write 
correct result objects; instead I would spend much stupid time in trying to 
construct by hand portions of ASTs. But I have just what I need already: normal 
runs of unitests in verbose mode write them out for me. Once I'm convinced, as 
much as possible, all works fine, I just copy paste result /strings/ into 
assert statements.
Then only I activate said assertions and switch verbose mode off, thus getting 
a regression test suite for free. Even better, this test suite holds 
information providing code, ready to use in case of failure.
Remedy: assert converts test result to string if needed.



Summary:

1. Named unittests allowing test suites in the form of (just an example):

unittest test1 {
     ...
}
unittest test2 {
     ...
}
unittest test3 {
     ...
}
unittest {
     test1;
     test2;
     test3;
}

/Unnamed/ unittests are run with --unittest. Named ones are intended to be 
called from unnamed ones. Backward compatible change.


2. A variant of assert(), or better a distinct check() statement, in the form of:

     check(expression, expectation)

Separating arguments allows writing out messages like (example formats):

* case success (outcome == expectation):
     <expression> --> <outcome>
bringing valuable feedback during development.

* case failure
     *** test failure **********
     expression  : <expression>
     outcome     : <outcome>
     expectation : <expectation>
     ***************************
In silent mode, check() writes only in case of failure.

If the expected result is a separated argument, it is trivial for check to call 
to!string on the outcome when it's not a string and the expected result is a 
string; then only compare.

I guess both changes require compiler support (but may very well be wrong).
As illustrated by the example, D does not prevent one to use such testing 
practices; instead, its very nice string and literal features make it far 
better suited for that than any other static language I know.
Still, like Walter, I think it's important for (good) testing practices to be 
supported by the language, so-to-say officially. 'unittest' itself does not 
bring any feature: one could trivially have a 'test' control func intended to 
be (un)commented. But the fact it's a builtin feature makes a decisive 
difference; if only because it then becomes part of the D community's shared 
culture.


Denis


(*) Please don't argue on this point, I have been a maintenance engineer for 
five years, diagnosing broken machines everyday.


=== sample part of test code ====================
void testList () {
     writeln("=== List ===============================================");
     List list;
     LexemeStream lexemes;

     // test data
     string[] sources = [
         `[  ]` ,
         `[ 3    ]` ,
         `[ 3   1  2 ]` ,
         `[zxy  x   zx]` ,
         `[nil false  true]` ,
         `["a"  "ab"   "abc"]` ,
         `[[3] [3 2] [3 1 2]]` ,
         `[3 "abc" zxy nil [3 1 2]]` ,
     ];
     string[] correctResultStrings = [
         `[]` ,
         `[3]` ,
         `[3 1 2]` ,
         `[zxy x zx]` ,
         `[nil false true]` ,
         `["a" "ab" "abc"]` ,
         `[[3] [3 2] [3 1 2]]` ,
         `[3 "abc" zxy nil [3 1 2]]` ,
     ];

     // test run
     foreach (i,source ; sources) {
         lexemes = claroLexer.lexemes(source);
         list = cast(List)matchList(lexemes);
         // output
         writefln("`%s` --> %s", source,list);
         // for diagnose
         if ( to!string(list) != correctResultStrings[i] ) {
             writefln("*** error *** expected: %s", correctResultStrings[i]);
             stop(); // core.stdc.stdlib.exit(0)
         }
         // regression test assertion
         assert ( to!string(list) == correctResultStrings[i] );
     }
     writeln();
}
...
unittest {
     testList();
//~     testUnit();
//~     testStatement();
//~     ...
}
=================================================
=== success output in 'verbose' mode ============
`[  ]` --> []
`[ 3    ]` --> [3]
`[ 3   1  2 ]` --> [3 1 2]
`[zxy  x   zx]` --> [zxy x zx]
`[nil false  true]` --> [nil false true]
`["a"  "ab"   "abc"]` --> ["a" "ab" "abc"]
`[[3] [3 2] [3 1 2]]` --> [[3] [3 2] [3 1 2]]
`[3 "abc" zxy nil [3 1 2]]` --> [3 "abc" zxy nil [3 1 2]]
=================================================

-- 
_________________
vita es estrany
spir.wikidot.com