[dmd-internals] changeset 455

[Jason House]

On Apr 28, 2010, at 5:45 PM, Andrei Alexandrescu <andrei at erdani.com>  
wrote:

> The problem is scale. Phobos is pretty large already. I did make  
> changes in unittests as you suggest above, but there are a lot of  
> those to make. Besides, when compiling Phobos as one unit and  
> unittesting everything, all unittests are run in one shot so you  
> don't know about modules and stuff.
>
> I improved that by painstakingly changing the build process to print  
> and unittest one Phobos module at a time, while unittests are  
> disabled in all other modules. That does help but is still far from  
> perfect.

I think there will be a lot of opinions on what should be done:
  • print failing test(s)
  • Module-level status like what you did
  • Test-level status (test name + pass/fail) like someone else  
suggested
  • Test with other log messages
  • Level based logging for customizable post-mortem review

As a reference, cruise control with nunit outputs a set of web pages.  
One is a top level showing each time the test suite was run and an  
overall success/failure indicator. Clicking on one brings up a summary  
page with number of tests, number passing tests, number of failed  
tests, number of skipped tests.  Further down is a summary of tests by  
category. Each one is colored based on %success, %failure, and  
%skipped. Clicking on one shows individual test results with failures  
first and skips second. Clicking on one of those tests shows the logs  
generated while running the test.

NUnit has a concept of test fixture setup, test setup, test teardown,  
and test fixture teardown. A test fixture with N tests would run test  
fixture and teardown once, but run test setup and teardown either 0 or  
N times. Test fixture setup failure skips all N tests. Test setup  
failure skips that one test.

I think providing overidable hooks for "starting test" and "finished  
test" is a 90% solution to letting people do whatever they deem best.