Let's improve the dmd tester.

[Jacob Carlborg]

On 2020-06-25 16:39, Stefan Koch wrote:
> Lately,
> 
> I have been working quite a lot on DMD again.
> (Thanks to my Employer who is using my experience).
> 
> One thing which makes intrusive changes to DMD, hard to do, is the 
> testing "script".
> Which executes and evaluates tests for DMD.
> 
> It greets you with a wall of text, in which passing and failing tests 
> are happily coexisting.
> Which means that seeing which test failed for example on the output of 
> Ci service, in which you can't do a grep, is a multi-second barrier in 
> ones development process.
> 
> I do believe that fixing that, will have a large positive impact on the 
> stability and maintainability of dmd.
> Which is important since it's still the main semantic engine behind D.
> 
> I am going to try and fix the tester, but help and suggestions are 
> always welcome.

The tests which are located in test/unit have a separate test runner 
(invoked by the main one) which only prints a summary when all tests are 
successful. You can invoke them explicitly using `test/run.d -u`:

$ ./test/run.d -u
unit_test_runner is already up-to-date
24 tests, 0 failures

It has several advantages to the other test runner:

* Compiles all tests into one executable, this will result in faster 
execution times

* Since all tests are compiled into one executable it's much more 
scalable then the main test runner. One can freely create directories 
and files to organize the tests without slowing it down too much

* Since the tests and the compiler runs in the same executable the tests 
have access to the compiler internals, make it much easier to test 
certain parts. For example, a test for the lexer does only need to 
invoke the lexer, not the whole compiler including code generation

* All tests have a UDA which describes what each test does

* It's possible to filter tests by passing only the files you would like 
to test:

$ ./test/run.d -u test/unit/deinitialization.d
unit_test_runner is already up-to-date
7 tests, 0 failures

* It's possible to filter tests by UDA:

$ ./test/run.d -u --filter Expression.deinitialize
unit_test_runner is already up-to-date
1 tests, 0 failures

* When a test fails it prints a nice output, including stacktrace, the 
UDAs attach to the tests and a summary of the failing tests:

$ ./test/run.d -u test/unit/deinitialization.d
unit_test_runner is already up-to-date
Failures:

1) global.deinitialize
core.exception.AssertError@/Users/doob/development/d/dlang/dmd/test/unit/deinitialization.d(28): 
unittest failure
----------------
??:? _d_unittestp [0x10f0687a1]
/Users/doob/development/d/dlang/dmd/test/unit/deinitialization.d:28 void 
deinitialization.__unittest_L4_C1() [0x10edf4380]
/Users/doob/development/d/dlang/dmd/test/unit/deinitialization.d:3 
core.runtime.UnitTestResult runner.unitTestRunner() [0x10ee02b57]
??:? runModuleUnitTests [0x10f069182]
??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int 
function(char[][])*).runAll() [0x10f08436c]
??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int 
function(char[][])*).tryExec(scope void delegate()) [0x10f0842f8]
??:? _d_run_main2 [0x10f08425c]
??:? _d_run_main [0x10f083fd9]
__main.d:1 main [0x10edf1e1d]
??:? start [0x7fff5f1473d4]

7 tests, 1 failures

Failed tests:
/Users/doob/development/d/dlang/dmd/test/unit/deinitialization.d:28

I recommend all tests in "tests/compilable" and "tests/fail_compilation" 
to be converted to the style of tests in "tests/unit".

-- 
/Jacob Carlborg