How can one reliably run unittests

[jfondren]

On Friday, 27 August 2021 at 10:33:02 UTC, Johan wrote:
> On Friday, 27 August 2021 at 07:48:40 UTC, Mathias LANG wrote:
>>
>> But when I throw in the `-unittest` switch (or the `-cov` or 
>> `-profile` for that matter), I want all the bells and 
>> whistles. Give me colors, an ncurses interface, an HTTP server 
>> even! Give me the full battery pack. Don't expect everyone and 
>> their mother to implement their own testing framework, that's 
>> just bonkers.
>
> Why would you expect a _compiler_ to implement a testing 
> framework?
> If the compiler implements the testing framework, any 
> improvement of the testing framework would require an update of 
> the compiler,

It makes for a better default experience, before someone's picked 
a testing framework and before code's gotten much investment. 
unittest{} is such a lightweight feature that a single module 
with a single function may as well have a unittest{} block.

Not too long ago dmd was completely silent on a successful 
-unittest run. Now it says "1 modules passed unittests" at the 
end. This improved the default experience and no proper testing 
frameworks were harmed.

> I think it's much better if the testing framework is 
> implementated separate from anything else.

That's already the case. silly, "The better test runner for D", 
is currently the #1 most popular package on 
https://code.dlang.org/ and has been in the top five for as long 
as I can remember.

It checks off everyone's complaints in this thread:
1. don't want to care about main? for historical reasons you 
still have to version(unittest) it out, but as dub is updated 
that won't be necessary either.
2. want colors? It has red and green unicode symbols.
3. don't want unittests{} blocks cluttering your code? put them 
in their own modules under tests/ and add "tests/*" to 
excludedSourceFiles of non-unittest configurations.
4. with minimal configuration following the README.md, all you 
have to do is run 'dub test' and add tests over time. It's not 
the case that everyone needs to write their own testing framework.
5. it doesn't bloat the normal build. (if you don't move the 
dependency on silly into your unittest configuration, the empty 
module adds 88 bytes to your normal build).

silly's another good example of hooking into the runtime's 
testing support: 
https://gitlab.com/AntonMeep/silly/-/blob/master/silly.d#L29

> That doesn't mean everybody has to implement it from scratch, 
> just that they should download it from some other location than 
> the compiler website.
> Separation of concerns.
>
> (by the way, I'm convinced `unittest` is a misfeature of the 
> language. Besides the clear downside that is apparent from this 
> thread,

What clear downside is that? A minor nit about -main? That 
there's a discussion happening at all?

I fixed the betterC bug that I noticed earlier in 
https://github.com/dlang/dmd/pull/13025 and progress is being 
made on deadalnix's complaint in 
https://github.com/dlang/dmd/pull/13026 , and for years people 
have been running 'dub test' without worrying about this or 
running 'dmd -unittest -run module', oops, 'dmd -main -unittest 
-run module', and getting more of the very gentle on-ramp to unit 
tests than they ever lost to the oops.

For the usual internet reasons this discussion is occasionally 
about 15x as heated as warranted, but since there's no fixing 
that, you shouldn't sweat it either.