Named unittests

[Johannes Pfau]

Am Sat, 18 May 2019 13:55:39 -0400 schrieb Andrei Alexandrescu:

> 
> Wait, if you build a program with -unittest will it run some/all of
> phobos' unittests? That would be indeed undesirable!
> 
> I wonder how often people compile external libraries together with the
> application within the same command line.

It runs some of the phobos unittests, those which are in templates 
instantiated in user code:
-------------------------------------------------
module phobos;
struct Foo(T)
{
    unittest {import std.stdio; writeln("Test");}
}

module user;
import phobos;
void main(){Foo!int a;}
-------------------------------------------------
dmd -unittest user.d

If you remove the Foo template parameter, the test will no longer run. 
The reason for this behavior is simple: We can only run the test for 
template instances, not for declarations.

The compiler could easily check whether the template declaration of a 
unittest in a template instance is part of the root modules (modules 
present on DMD command line) and skip other tests. However, this will 
break the test patterns where people intentionally instantiate templates 
in other modules. Tests in the same module do not cover visibility issues 
and especially if we ever get proper DLL support for windows, where we 
would have to mark functions explicitly as export, you really do want to 
test in another module, likely even in another library.




I think the biggest problem with D's unittest implementation is that it 
isn't built on small, orthogonal and composable features. Instead we 
collect all unittests in a module, generate a function which calls them 
one by one and the add a pointer to that function to ModuleInfo. This 
implementation is highly specific and very inflexible: With the current 
framework, we simply cannot continue running tests after one failed, as 
the compiler essentially concatenated all of them into one function.

I posted a DMD PR which allows running tests individually seven/five 
years ago:
https://github.com/dlang/dmd/pull/1131 https://github.com/dlang/dmd/pull/
3518
Back then this was shot down on details whether we should allow to 
disable unittests and whether we should add file/line information. With 
the current implementation (storing information in ModuleInfo), this is 
something we as compiler developers have to decide on. But all these 
issues are detail decisions, which should really be left to unittest 
library/runner authors. For the core language/compiler we should rather 
provide reusable building blocks which are flexible enough to allow 
different library implementations instead of a highly specific, 
inflexible implementation.



So why do we have this specific compiler implementation? What does it do 
that library code can't do? Basically it does two things:

1) Test discovery: Decide which tests should be run and collect all of 
them. The selection criteria is essentially this: Unittests in files 
passed to dmd like this: 'dmd -unittest file.d' are selected. Also some 
tests in other modules because of the template problem. 

2) Test registration: For each discovered tests, make it somehow possible 
for the runtime to run it.


Now we do have library test runners*. They use compile time reflection to 
find tests, build on UDAs and other well-known language features and I 
really think this is the way to go. In order to switch druntime to such 
an implementation, we would need to reach feature parity with the current 
implementation though. The main problem here is in 1), as reflection 
based runners require you to somehow list all modules which should be 
tested.
I do not think we want to mess with ModuleInfo here, as this always 
involves serializing compile-time type information to runtime 
information. We want a completely compile time solution here. So what we 
need is some way for a library to reflect on every other application 
module automatically. I'd propose to leverage template mixins for this 
and extend them in one crucial point. Template mixins almost do what we 
want:


-------------------------------------------------
module test;

void registerTest(string name)
{
    import std.stdio;
    writeln(name);
}

mixin template registerMixin(string name)
{
    pragma(crt_constructor)
    extern(C) void registerUnittest()
    //FIXME: Need a unique mangle for this function
    {
        mixin("alias members = __traits(allMembers, " ~ name ~ ");");
        foreach(member; members)
            registerTest(member.stringof);
    }
}
-------------------------------------------------
module user;
import test;

mixin registerMixin!"user";

void main() {}
-------------------------------------------------
dmd main.d test.d


The only problem here is that we have to manually invoke `mixin 
registerMixin!"user";` in every module we want to test. Now what if we add 
`import mixins`? Simply change the `registerMixin` signature to `import 
mixin template registerMixin(string name)` and whenever the module is 
imported, the compiler automatically inserts the mixin line into the 
importing module. Essentially an import mixin template is a template 
automatically mixed in into importing modules and able to reflect on 
these modules. This small addition now allows us to use all the compile 
time, template based reflection stuff and have automatically registered 
tests. As this feature is generic we could also use it to auto-register 
@benchmark(UDA) functions. Or automatically generate serialization stuff. 
Or generate runtime typeinfo. Or .... And every library author can do the 
same.

I realize this feature is a bit controversial, as it essentially allows 
library authors to silently hook code into user modules. OTOH I think 
this is a very orthogonal feature enabling many new idioms to be explored.



2) is then easily solved, we already have crt_constructor. You could also 
place pointers into a named section for low level targets and there may 
be other options. Registration is basically a solved problem.



For the default druntime tester, we could just add such an import mixin 
template to object.d, so it's included in every file. Then wrap the code 
in the mixin template in version(unittest) and we're mostly done. Not 
including a main function / running the main application logic would 
likely be the users responsibility then, as a D library cannot really 
prevent the runtime from calling main. The remaining problem are tests in 
templates (see *), as we can't automatically register these without 
additional compiler help. For now, the compiler could explicity emit 
registerTest calls for these and we could add a deprecation for this, 
advising to explicitly test using RegisterTests!(T).


* Note that CTFE reflection based test runners do not run tests in 
templated aggregates, so they are not affected by the template issue. 
Actually, this seems to be a very good thing: You could easily create a 
template which runs nested tests on demand using reflection on the type:
RegisterTests!(Foo!int);
You can easily place this into different libraries, modules, files, ... 
and it will always work as expected.

-- 
Johannes