How can one reliably run unittests

[Alexandru Ermicioi]

On Tuesday, 31 August 2021 at 16:22:17 UTC, H. S. Teoh wrote:
> On Tue, Aug 31, 2021 at 11:08:46AM +0000, deadalnix via 
> Digitalmars-d wrote:
>> On Friday, 27 August 2021 at 14:13:56 UTC, Steven 
>> Schveighoffer wrote:
>> > separate compilation. The compiler doesn't know all the 
>> > modules that have been built with unittests. Only the 
>> > runtime does.
>> > 
>> 
>> The is another extremely weird design.
>> 
>> When you run the unitests, you want to run the unitests for 
>> the module you specifically asked for, not for half the galaxy.
>
> Yeah, this problem has been brought up before, but nobody had 
> the solution.
>
> The problem is that when you compile with -unittest, it turns 
> on unittests for ALL modules, including Phobos, 3rd party 
> libraries, *everything*.  This is rarely what you want -- 
> Phobos, for example, contains a bunch of extremely heavy duty 
> unittests that end users don't ever want to run.  Because of 
> this, the version=StdUnittest hack was implemented in Phobos.  
> But this is not scalable: every library and his neighbour's dog 
> would have to implement their own unittest hack version 
> identifier for this scheme to work.  And if some 3rd party 
> library author neglected to do this, you're left out in the 
> cold.
>
> I think it makes much more sense to only compile unittests for 
> modules explicitly specified on the command-line. After all, 
> the user is unlikely to be interested in unittesting 3rd party 
> libraries; his primary interest is to unittest his *own* code.  
> However, this simplistic approach falls flat in the face of 
> `dmd -i`.  Schemes like `-unittest=mod1,mod2,...` have been 
> proposed before, but AFAIK never implemented.
>
>
> T

Then perhaps have some exlude-unittest argument that accepts 
packages instead?
That would allow to exclude unittest from entire set of modules.

Also why do we even make an executable when we compile with 
-unittest flag?
Can't we just make a shared lib, and then have a unit test runner 
loading this lib and run the tests in it?
Imho this would be a lot more flexible. Dmd could come with an 
default runner, and if someone doesn't like it, he could just 
replace runner with his own implementation, that would do all the 
prettyfying he wants.

The only thing needed here is well built unit test api exposed to 
the runner and reflection api at runtime, to allow that said 
runner inspect underlying tests for additional information, such 
as annotations, to do more advanced logic than default runner 
would do.

Regards,
Alexandru.