Utah Valley University teaches D (using TDPL)

[Jonathan M Davis]

On Friday 26 November 2010 18:52:59 Walter Bright wrote:
> Jonathan M Davis wrote:
> > I'm a firm believer that D unit tests should not change how they
> > fundamentally work at this point. I don't _want_ it to report the number
> > of tests that passed.
> 
> That's right. The number that fail is completely useless window dressing.
> 
> > That information is not at all useful. And if I want to do something like
> > set up something to run dmd periodically and report only when the tests
> > start failing, then the current situation is perfect.
> > 
> > I _do_ think that the unit testing stuff should be expanded to have named
> > unit tests and do whatever is necessary to make it possible for external
> > tools to run the unit tests. Then you could have an external tool that
> > did things like only run specific unit tests and report which tests
> > succeeded and which failed.
> > 
> > So, I do think that it should be possible to build external tools on D's
> > unit testing framework, but I think that it's quite good as it is now,
> > and I wouldn't want to see it drastically changed.
> > 
> > The two changes that I want to see to the current framework are
> > 
> > 1. Make it so that every unittest block runs in a module instead of them
> > stopping once one failed. The current situation is better than stopping
> > _all_ unittest blocks once a single test fails, but it's still not
> > granular enough. I believe that this change is planned, but work has to
> > be done to make it possible, and that work hasn't been done yet.
> 
> I believe that is the current behavior. For a time, it kept getting broken
> by changes to druntime, so it may be broken yet again, but that is the way
> it is supposed to work.

It's not. This program

import std.stdio;

bool func()
{
    return false;
}

void main()
{
}

unittest
{
    writeln("test 1");
    assert(func());
}

unittest
{
    writeln("test 2");
}

prints

test 1
core.exception.AssertError at test(15): unittest failure
----------------
./test(onAssertErrorMsg+0x34) [0x808c0f4]
./test(onUnittestErrorMsg+0x18) [0x807f3b8]
./test(_d_unittestm+0x22) [0x807d252]
./test(void test.__unittest_fail(int)) [0x807acfa]
./test(void test.__unittest1()) [0x807ac3b]
./test(void test.__modtest()) [0x807ace0]
./test(extern (C) bool core.runtime.runModuleUnitTests()) [0x807f56c]
./test(_D6object10ModuleInfo7opApplyFMDFKPS6object10ModuleInfoZiZi+0x41) 
[0x807cd7d]
./test(runModuleUnitTests+0x87) [0x807f487]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x807d448]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x807d370]
./test(main+0x96) [0x807d316]
/usr/lib32/libc.so.6(__libc_start_main+0xe6) [0xf75b7c76]
./test() [0x807ab51]

The second unittest block is never run. I believe that the last time that this 
got discussed on the Phobos list, Sean said that some fundamental changes had to 
be made to either dmd or druntime (I _think_ it was dmd, but I'm not sure) to 
make it possible to run subsequent unit tests in a module after one of them has 
failed. If a test in a module fails, then the tests in other modules run, but 
not the rest of the tests in the module that had the failure.

> > 2. Make it possible to name unittest blocks. This would make exceptions
> > that escape unittest blocks _far_ more useful. Names like unittest45
> > really aren't useful at all. I can't even figure out how it does the
> > numbering.
> 
> If the line numbers given out are incorrect, those are compiler bugs and
> I'd like to get them fixed.

Not the line number. I'm talking about the function name that you get stack 
traces. Take this program for instance:

void func()
{
    throw new Exception("An exception threw.");
}

void main()
{
}

unittest
{
    func();
}

If you run the unit tests, you get

object.Exception: An exception threw.
----------------
./test(void test.func()) [0x80599fe]
./test(void test.__unittest1()) [0x8059a10]
./test(void test.__modtest()) [0x8059a1c]
./test(extern (C) bool core.runtime.runModuleUnitTests()) [0x805dcdc]
./test(_D6object10ModuleInfo7opApplyFMDFKPS6object10ModuleInfoZiZi+0x41) 
[0x805ba9d]
./test(runModuleUnitTests+0x87) [0x805dbf7]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x805c168]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x805c090]
./test(main+0x96) [0x805c036]
/usr/lib32/libc.so.6(__libc_start_main+0xe6) [0xf7595c76]
./test() [0x8059921]


Notic __unittest1(). Here, there's only one unit test, so it's not a big deal, 
but once you have a lot of them, it's not really feasible to associate that 
function name with a unittest block. If the unittest blocks were named, 
something like this

unitest(mytest)
{
}

then you could get a stack trace like this

object.Exception: An exception threw.
----------------
./test(void test.func()) [0x80599fe]
./test(void test.__unittest_mytest()) [0x8059a10]
./test(void test.__modtest()) [0x8059a1c]
./test(extern (C) bool core.runtime.runModuleUnitTests()) [0x805dcdc]
./test(_D6object10ModuleInfo7opApplyFMDFKPS6object10ModuleInfoZiZi+0x41) 
[0x805ba9d]
./test(runModuleUnitTests+0x87) [0x805dbf7]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x805c168]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x805c090]
./test(main+0x96) [0x805c036]
/usr/lib32/libc.so.6(__libc_start_main+0xe6) [0xf7595c76]
./test() [0x8059921]

With that stack trace, I can easily determine which test threw the exception. 
It's not a problem with assertions within the test - since they'll give the file 
and line number of the assertion in the test - but if any function called by the 
test throws an exception for whatever reason (including assertions in 
contracts), then you get a stack trace that borders on useless, because you 
can't easily determine which unittest block threw the exception. With named 
unittest blocks, that problem can be fixed.

As it stands, I don't even know how the number for the unittest is picked. It 
does not appear that __unittestX() necessarily corresponds to the Xth unit test 
in a module lexographically. Named unit tests are really the way that the 
problem should be fixed.

I'm not saying that naming unittest blocks should be mandatory, but I think that 
it should be possible. Unnamed unittest blocks result in functions with names 
like __unittest1() like they do now, whereas named unittest blocks would  result 
in names like __unittest_testname() where testname is the name of the unittest 
block. It would make stack traces far easier to deal with, and if external tools 
are ever going to have any hope of calling unit test functions, then I think 
that it's a necessity

>  > Also, having named unittest blocks would be a necessity for properly
>  > allowing
> 
> external tools to run the unit tests.
> 
> I don't really know how that would work.

I'm not sure either. But ideally, it would be possible to have an external tool 
run a program which runs all of the static constructors and whatever else has to 
be run before the unittest blocks can run, and then specifically run each 
unittest block that it wants to, in whatever order it wants to, without 
necessarily running all of them.

For instance, if you use JUnit with Eclipse, you can tell it to specifically run 
all of the unit tests in a file. Or you can tell it to run only a single unit 
test. Or you can tell it to run them all. Right now, with D, you can only ever 
run them all, and you can't really integrate them into an IDE. I think that that 
should become possible at some point. Unfortunately, however, I don't know 
enough to say exactly how that would work. I don't even know how it works with 
JUnit and Eclipse. It is potentially a very important feature though.

> > I do _not_ want to see D programs printing out anything more than they do
> > when running unit tests. I'm totally open to external tools which do
> > more (it would particularly good for IDEs to be able to run unittest
> > blocks individually), but I do not want to see the basic framework
> > change how it prints feedback on test successes and failures. It works
> > great as it is now.
> 
> Yes, and if anyone wants more, a writefln("your message here") works just
> fine.

Yes. It is a bit of work to make all of your unit tests print success or 
failure, but it's not hard, just tedious. I see no reason to make everyone have 
to put up with extraneous output just because some people want it to print a 
list of successes and failures. If you really want that, you can do it yourself. 
The one part I can think of that you can't do yourself very well is print the 
total number of successes and failures (unless you print a running total with 
each test). But it's possible to build printing functionality on top of a 
framework that only prints out on failure, whereas it's not possible to get rid 
of printing with a framework that always prints.

- Jonathan M Davis