About unittest, test runners and assert libraries

[Zardoz]

I like to talk about the state of unit testing, test runner & 
frameworks and assert libraries, as I did some little experiments 
recently.

DLang, put a strong emphasis on unit testing. Language & tooling 
support for unit testing (unittest blocks, assert, assert helpers 
on std.exception and AssertError on core.exception, dub test ...)

However, the out of the box test runner embedded on the guts of 
DLang, it's pretty limited. So, many test runner has been write, 
as can bee see on dub package repository. This by itself, isn't 
bad. Having several test runner to choose it's good. Specially 
when one test runner it's specialized on some ways that other 
not.. being faster or being straightforward to use, make more 
easy to integrate with IDE, etc. Some of these test runners, are 
really more a test framework, as include some helper libraries 
for mocking & stubing, fluent asserts, bdd, etc.

And separated of the test runners, we have some auxiliary 
libraries, that implements mocking, fluent asserts, etc. Aiming 
to allow to be used on agnostic way respect the test runner, or 
even on the default test runner that comes with DLang.

So, I like to show what I saw trying the five most popular test 
runners (Unit-threaded, Silly, Trial, DUnit and D-Unit), and a 
little details that puzzles me about the different assert 
libraries.

### D-Unit

D-Unit try to be a simple implementation of the xUnit Testing 
Framework. It complete ignores unittest blocks and requires to 
encapsulate the tests on a class, following the same pattern that 
does xUnit/jUnit (but using a mixin, instead of extending from 
some base class). Includes some assert helpers following the 
xUnit/jUnit framework, etc. Even can generate xUnit XML report 
files and differentiated between a failed test (ie. a test case 
where an assertion has failed) from an errored test (ie. a test 
where a uncaught exception has been raised)

However, isn't straightforward to use (dub run -c 
*my-d-unit-config*), and only understand as failed test only when 
it's being using his own assert helper functions. A failed assert 
form any other 3rd party assert library will be marked as an 
error.

### DUnit

DUnit tries to being another full framework with mocking & assert 
helpers.

Sadly, it's dead. The **githup repo it's archived*, and don't see 
any update since early 2020's. Also, isn't very straightforward 
to use (requires some extra config on dub.json, but at least 
could be run with a simple dub test). Other annoying thing, it's 
that the runner iot's pretty limited. Can't list the tests, run a 
single test, or show anything useful if the test run OK. Also, 
the pretty print of a error/fail on a test it's implemented on 
his own assert helpers. Using 3rd party assert library on it, 
would be the same experience that using on the DLang embed test 
runner.

### Trial

Trial aims to be a very powerful test runner. It allow to use a 
config file to select the reporters (in plural), how discover 
tests, policies, and test runner implementation. Also, aims to be 
more easy to integrate with IDEs. Even the author had a Visual 
Code extension that auto uses Trial to discover tests on a 
project. Sadly this extension it's dead, as the author saw zero 
interest on it.

Using directly Trial, it's really weird. I couldn't find a way to 
allow it to be used as dub test or dub run on the project. 
Instead, you must build the Trial executable (dub run trial does 
the work), and it will find&execute the test in your project. 
Also, on my case, only worked using the master branch. Because 
Trial uses internally dub (compiles dub in it'self) and so, it's 
strongly coupled with dub.

It's really sad about the Visual Code extension, because gives 
the quick&easiest experience to run & debug tests that I saw 
using DLang on a IDE.

### Silly

Silly tries to being a better test runner that it's simple to 
use, could run test on multiple threads, color output, 
list&filter tests.

To use, it's really the most easy & straightforward of any test 
runner that I try. I simply need to add a dependency to Silly, 
and run dub test. If it's being used on a library, it's 
recommended to add the dependency on a on "unittest" config or 
use it on a sub package or separated dub.json .

In my personal opinion, it's the best simply because it's the 
most easy to use. And don't give headhaches to configure or 
require special stuff to discover tests, or put tests inside 
classes, etc. Gives a quick&short summary, and gives good 
information when a test fails/errors.

However, there it's room for improvement seeing what Trial and 
D-Unit could do. Also looks that the author it's a bit absent. 
Plus, there is some issues on the gitlab repository...

### Unit-threaded

Unit-threaded shares some goodies with Silly. Like his name 
pinpoints, run the tests on multiple threads. Also, aims to being 
a testing framework, including hos own assertions library, 
mocking helpers, sandboxing filesystem, integration testing 
facilities...

However, to discover tests, requires manually generating a file 
to register the tests or add a prebuildcommand to dub.json to 
autogenerate it. Something very weird, when the other tests 
runners can discover the test on the fly. Also, this make more 
problematic run some tests only when a dub configuration it's 
activated.

Another weird thing, it's that expects that 3rd party assert 
libraries, use/extend from his own exception (UnitTestException) 
to fail on a test. If not, it show not useful junk stack trace on 
the output.


### About Asserts, assert libraries and AssertError

And this last details, makes my head to scratch, because I saw 
this on many assert libraries. Every assert library, throws a 
plain Exception (or UnitTestException if Unit-threaded it's added 
as dependency). This make really impossible to differentiate a 
failed test from an errored test (Remember, an test case where an 
assertion fails, its a fail test. An test case where a uncaught 
exception it's raised its an error.)

To me, sound logic that any assert library should use the same 
exception that it's throw by DLang's assert(). This is 
AssertError. And that tests runners should count any AssertError 
(or derived) inside a test case as failed, and any other 
exception as an error. This allows to assert libraries and assert 
runners work better without needing to be coupled to extend from 
some internal Exception implemented by a test runner and to have 
code on assert libraries to handle every test runner on the wild.

So, I like to ask **WHY no body does this**. Perhaps i don't see 
the inconvenient. It's a thing that puzzles me, and I think that 
should be fixed. It's one of many rought edged that have DLang 
and that it's pretty to fix.

I have my little experiment fork/branch of Silly doing exactly 
this, and Pijamas develop branch using AssertError. The result, 
it's that Silly counts correctly failed and errored tests and 
Silly & Pijamas not have any coupling between.

PD :  Sorry for my poor english. Also, I have a strong background 
with Java and JavaScript, so I have tendency to use jUnit, Spock, 
jShould, etc... as some kind of gold standard for how unit 
testing should be.