Named unittests

[Jacob Carlborg via Digitalmars-d]

On 2015-03-30 23:52, Andrei Alexandrescu wrote:
> We're having a strong need for named unittests at Facebook for multiple
> reasons.
>
> 1. We have sophisticated tooling that verifies whether unittests are
> flaky. The automated monitor (for e.g. C++) figures whether a given
> unittest fails several times across several commits. Unittests are
> identified by name; relying on file/line is impossible because the line
> of a failure is not stable across changes.
>
> 2. Again for efficient automated testing and flakiness detection, one
> should be able to run only a subset of unittests by mentioning them by
> line in the command line. Note that this implies there's no
> interdependency between distinct unittests, which is fine because the
> new ability is opt-on; I'd say is pure style anyway.
>
> 3. Mentioning unittest names in failure messages helps human
> communication (e.g. "AddPeer is failing after your change"). This is
> impossible with file and line numbers.
>
> I'd like to make a DIP for named unittests. Who can help me with that?

I completely agree. I always thought that the built-in unit test support 
wasn't sufficient. All of the above should be possible to implement in 
library code without needing to change the language.

Now, it depends on how far you want to go. We could do something simple 
as adding a UDA, which has already been suggested by others:

@name("this is a test") unittest
{
     assert(false);
}

Personally I would like a complete testing framework like RSpec. I have 
a very simple implementation [1] of this:

unittest
{
   describe("std.uni", {
     describe("toUpper", {
       it("converts a string to uppercase", {
         "foo".toUpper.should.eq("FOO")
       });
     });

     describe("toLower", {
       it("converts a string to lowercase", {
         "Foo".toLower.should.eq("foo")
       });
     });
   });
}

With RSpec, which supports multiple formatters, it can look like this:

---
Randomized with seed 28149

std.uni
   toUpper
     converts a string to uppercase
   toLower
     converts a string to lowercase

Finished in 0.00078 seconds (files took 0.0931 seconds to load)
2 examples, 0 failures

Randomized with seed 28149
---

This shows a failing test:

---
Randomized with seed 57730

std.uni
   toLower
     converts a string to lowercase
   toUpper
     converts a string to uppercase (FAILED - 1)

Failures:

   1) std.uni toUpper converts a string to uppercase
      Failure/Error: 'foo'.should == 'FOO'
        expected: "FOO"
             got: "foo" (using ==)
      # ./spec/foo_spec.rb:6:in `block (3 levels) in <top (required)>'

Finished in 0.00298 seconds (files took 0.08553 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./spec/foo_spec.rb:5 # std.uni toUpper converts a string to uppercase

Randomized with seed 57730
---

Or with the TextMate formatter [2]. It shows both passing and failing 
test. When a test fails to get a link pointing back to the editor and a 
syntax highlighted snippet of the failing source code.

With RSpec it's also possible to specify the --line-number flag which 
will only run the tests matching a given line number. Or the --example 
flag which will run all examples (tests) matching the given string.

Is any of this interesting to have in Phobos? Otherwise I'll continue 
working on my own framework.

[1] https://github.com/jacob-carlborg/dspec
[2] http://thejqr.com/2009/02/06/textmate-rspec-and-dot-spec-party.html

-- 
/Jacob Carlborg