Helper unit testing functions in Phobos (possible std.unittests)

[Jonathan M Davis]

On Friday 05 November 2010 22:54:20 Nick Sabalausky wrote:
> "Jonathan M Davis" <jmdavisProg at gmx.com> wrote in message
> news:mailman.125.1289018234.21107.digitalmars-d at puremagic.com...
> 
> > I'm proposing a possible new module for phobos which would be called
> > std.unittests. The code and DDoc file can be found here:
> > http://is.gd/gLH9Q
> > 
> > Or you can look at the code here directly (though it has poor
> > highlighting):
> > http://ideone.com/EOlod
> > 
> > The module contains a set of helper functions for unit tests which are an
> > improvement over simply using assert. I have them to be very helpful in
> > the unit
> > tests in my proposal for std.datetime. My unit tests are more concise,
> > easier to
> > read, and much easier to debug because of them. assertEqual() and
> > assertExcThrown() have been particularly useful.
> > 
> > assert is a great tool, but simply telling you that an assertion failed
> > is not
> > as useful as telling you _how_ it failed, and a function like
> > assertExcThrown()
> > (which, as you may guess, asserts that a particular exception was thrown
> > by a
> > particular function call) really reduces the boiler plate code that you
> > have to
> > write for testing what it tests. So, I find such helper functions are
> > extremely
> > useful.
> > 
> > Because I have used them so extensively in std.datetime, they are pretty
> > much
> > either going to have to be added as a module to Phobos or added as
> > private helper functions in std.datetime. Personally,  I think that they
> > (or something
> > very similar to them) merit being added as a new module in Phobos, which
> > can be
> > expanded upon as others come up with particularly useful unit testing
> > functions.
> > 
> > So, I'm requesting that the folks here on the list have a look at my
> > module and
> > review it for possible inclusion in Phobos. I don't know what Andrei's
> > position
> > on them is (he rightly focused his review of my datetime code on the
> > date/time
> > functionality and API rather than its unit tests and didn't mention my
> > unit
> > testing functions at all), but I think that such a module would be of
> > great
> > worth and would like other's opinions on it.
> 
> I made a very similar thing in my SemiTwist D Tools library, so I can
> definitely attest to D's need for all of this.
> 
> My comments on this one, based on a breif skim of the doc:
> 
> I would try to change it so this:
> 
> assertOpCmp!("<")(myfunc(), 7, "My test failed!");
> 
> Can be written more like these:
> 
> assert!("_ < 7", "myfunc()", "My test failed!")();
> assert!("_1 < _2", "myfunc()", "7", "My test failed!")();
> assert!("_1 < _2", "myfunc()", "somethingElse()", "My test failed!")();
> 
> Aside from being more flexible, this way when it fails it can report much
> more relevent information that can actually compete with JUnit/NUnit. For
> instance, even with the custom message omitted, it could still give info
> like:
> 
> Failed: 'myfunc() < 7'
> 'myfunc()' was '13'

I don't really see the benefit of this. As it is, assertOpComp!() clearly tells 
you what the actual result was and what the values were. It also reports the file 
and line number of the failed test. What more do you need? Seeing the exact 
expressions that resulted in the values to be tested doesn't strike me as 
necessary. You have the file and line number. You know what the exact result was. 
You can look at that exact line to see what produced the values that were 
tested. It seems to me like you have all of the information you need.

> Also, I'd say if file/line info can't be inferred automatically because of
> varargs, then the interface needs to lose the vararg ability. It'd be nice
> to have, but not needing to provide line/file info manually is much nicer.
> Although, we could probably have both by using the suggestion above (ie,
> make the whole thing generate a string to be mixed-in).

We need the ability to call arbitrary functions with some of the unit test 
functions (the ones where you have to pass LineInfo()). I don't see how you 
could do that with a string mixin unless you made the test itself a mixin 
instead of a function call, which strikes me as quite unpleasant. I'm certainly 
up for suggestions, but what I have is the best solution that I've seen. It is 
rather annoying to have to pass LineInfo() as the first argument, but it's 
straightforward. It works. And it's not like you can forget to do it, since it 
won't compile if you do.

> Do the 'assertEqual', 'assertOpCmp', etc, all report if an exception was
> thrown (and if so, what exception)? If not, they should.

They throw just like assert would. Even assertExcThrown() throws the exception 
if it's not the type of exception that you were testing for. I see no benefit in 
having them catch stray exceptions. That's not what they're testing for, and I 
believe that just like with assert, if a unit test function fails, that should 
be the end of that unit test. The exception gets reported just as it would be 
with an assert that threw an exception. So, it seems to me to be the correct 
behavior. The tests test for exactly what you tell them to and let other 
exceptions through to be reported by the runtime.

- Jonathan M Davis