How can one reliably run unittests

[Steven Schveighoffer]

On 8/31/21 12:22 PM, H. S. Teoh wrote:
> On Tue, Aug 31, 2021 at 11:08:46AM +0000, deadalnix via Digitalmars-d wrote:
>> On Friday, 27 August 2021 at 14:13:56 UTC, Steven Schveighoffer wrote:
>>> separate compilation. The compiler doesn't know all the modules that
>>> have been built with unittests. Only the runtime does.
>>>
>>
>> The is another extremely weird design.
>>
>> When you run the unitests, you want to run the unitests for the module
>> you specifically asked for, not for half the galaxy.

This is exactly how it works. You ask to build unittests for a module by 
specifying `-unittest` on the build line for that module. The linker 
hooks up the module infos into a data segment, and then the runtime 
looks through those looking for ones with unittests to run.

So for instance, if you link against library X, and you didn't actually 
build library X with -unittest, then you won't run library X's 
unittests, only the ones in your project.

But my point is, the reason it's done this way is because D uses the 
build system that C uses -- which has a compiler and a linker as 
separate steps.

It's also the reason that cycle detection can't be run until program 
startup.

> 
> Yeah, this problem has been brought up before, but nobody had the
> solution.
> 
> The problem is that when you compile with -unittest, it turns on
> unittests for ALL modules, including Phobos, 3rd party libraries,
> *everything*.  This is rarely what you want -- Phobos, for example,
> contains a bunch of extremely heavy duty unittests that end users don't
> ever want to run.  Because of this, the version=StdUnittest hack was
> implemented in Phobos.  But this is not scalable: every library and his
> neighbour's dog would have to implement their own unittest hack version
> identifier for this scheme to work.  And if some 3rd party library
> author neglected to do this, you're left out in the cold.

There is some misunderstandings with what you say here.

When you compile with `-unittest`, all unittests are turned on BUT only 
unittests in modules given on the command line are semantically analyzed 
and put into object files. So you don't get all the unittests from 3rd 
party libraries in your build. However, it does turn on the 
`version(unittest)`, which might make some libraries alter how their 
types are defined.

It also will include unittests inside templates that you instantiate 
(because it doesn't know if those were already done somewhere). This is 
why I very much dislike unittests inside templates.

The `version(StdUnittest)` was added so that extra imports, altered and 
extra types were not included, as well as removing unittests inside 
templates. It wasn't added to exclude standard unittests (which was 
already happening).

-Steve