How can one reliably run unittests

[Steven Schveighoffer]

On 9/2/21 1:26 PM, deadalnix wrote:
> On Tuesday, 31 August 2021 at 16:50:07 UTC, Steven Schveighoffer wrote:
>> So for instance, if you link against library X, and you didn't 
>> actually build library X with -unittest, then you won't run library 
>> X's unittests, only the ones in your project.
>>
> 
> So I was curious at to what was meant by this. Let me try to make it 
> explicit why the current behavior is nonsense. I ran something akin to:
> 
> ```
> $ dmd -m64 -Isrc -w -debug -g -main -unittest -i -run src/somemodule.d
> 3 modules passed unittests
> ```
> 
> It is clear that this is running random unitests I don't care about. 
> Module that are brought in via the -i flag are also compiled with their 
> unittests, which is nonsense.

The -i flag finds all dependencies and builds them all at once as if you 
weren't linking in the object code that is in a library. That's not what 
I was saying.

What I was saying is, you have a library, you built it without -unittest 
(on its own build line, or just use dub because it does the right 
thing), and then you build your main application with -unittest, linking 
in the already-built library (or its object files). The runtime runs all 
unittests it finds, and so it's running just the modules that were built 
with the -unittest flag.

When you use -i, it means *ALL MODULES AND IMPORTS* are built with the 
-unittest flag, and so they all run.

> 
> Will run exclusively the tests for module1.d and ignore module2.d , 
> which is considered an argument to be passed to the main that don't 
> exist and is never run.
> 
> This is complete nonsense (and means that I simply run a fraction of my 
> unitests and have no idea how to fix my build to runt hem all).

-i is going to run all unittests. How else would it interpret the 
command line? If you want to build some modules differently than others
(i.e. some with unittests and some without), you can't use -i.

>> But my point is, the reason it's done this way is because D uses the 
>> build system that C uses -- which has a compiler and a linker as 
>> separate steps.
>>
> 
> I don't think this is a good reason. There is nothing in the C style 
> build.link model that prevent us from doing the right thing.

What is the "right thing"? Reading what you expect in your mind is not 
what a compiler can do. Define how it will work. I think unittests work 
acceptably the way they are now, but I'm sure others would be keen to 
have some more bells and whistles. Just make a proposal and get it 
merged in.

Start by saying "when I type `dmd -m64 -Isrc -w -debug -g -main 
-unittest -i -run src/somemodule.d`, here is what I expect to happen: 
...", and then define rules that will make that happen.

-Steve