Suggestion (ping Walter): Improve unit testing.

[Gregor Richards]

There are a number of improvements I'd like to see in D's unit testing 
framework. All of them are standard library changes, and I've made them 
work in Tango (as an external patch). If they were added to both Phobos 
and Tango, DSSS could easily run unit tests, which would improve its 
test suite.

Here are two major problems I see with unit tests right now:
1) You can't run only unit tests, you are forced to run main() as well.
2) Unit testing doesn't actually output anything useful, it just works 
under the no-news-is-good-news principle.

Each problem has a fairly easy solution in the core library:



1) Adding a special option, handled by the core library's main function, 
which will cause only unit tests to run. The changes I made in Tango:
Index: lib/compiler/gdc/dgccmain2.d
===================================================================
--- lib/compiler/gdc/dgccmain2.d	(revision 2100)
+++ lib/compiler/gdc/dgccmain2.d	(working copy)
@@ -35,7 +35,7 @@
  extern (C) void _minit();
  extern (C) void _moduleCtor();
  extern (C) void _moduleDtor();
-extern (C) void _moduleUnitTests();
+extern (C) void _moduleUnitTests(int);

  /***********************************
   * These functions must be defined for any D program linked
@@ -108,6 +108,7 @@
  {
      char[][] args;
      int result;
+    int testOnly = 0;

      version (GC_Use_Stack_Guess)
      {
@@ -137,11 +138,17 @@
          }
          args = am[0 .. argc];
      }
+
+    // check for --d-unittests-only
+    foreach (arg; args) {
+        if (arg == "--d-unittests-only")
+            testOnly = 1;
+    }

      void run()
      {
          _moduleCtor();
-        _moduleUnitTests();
+        _moduleUnitTests(testOnly);
          result = main_func(args);
          isHalting = true;
          _moduleDtor();


As you can see, if --d-unittests-only is supplied, it passes a 1 into 
the (now modified) _moduleUnitTests function, which will cause it to 
exit when finished testing.



2) The _moduleUnitTests function can be improved to provide useful 
output. The output I've implemented for Tango is:
Failure in test for module 'test':
   tango.core.Exception.AssertException on test.d(7): Assertion failure
TESTS: 1  PASSED: 0  FAILED: 1

The patch to make this work was fairly simple as well:
Index: lib/compiler/gdc/genobj.d
===================================================================
--- lib/compiler/gdc/genobj.d	(revision 2100)
+++ lib/compiler/gdc/genobj.d	(working copy)
@@ -42,7 +42,8 @@
      import tango.stdc.string; // : memcmp, memcpy;
      import tango.stdc.stdlib; // : calloc, realloc, free;
      import util.string;
-    debug(PRINTF) import tango.stdc.stdio; // : printf;
+    //debug(PRINTF) import tango.stdc.stdio; // : printf;
+    extern (C) int printf(char*, ...);

      extern (C) void onOutOfMemoryError();
      extern (C) Object _d_newclass(ClassInfo ci);
@@ -1054,11 +1055,13 @@
  }

  /**
- * Run unit tests.
+ * Run unit tests. If testOnly is true, output results and quit.
   */

-extern (C) void _moduleUnitTests()
+extern (C) void _moduleUnitTests(int testOnly)
  {
+    int testCount, testFail;
+    testCount = testFail = 0;
      debug(PRINTF) printf("_moduleUnitTests()\n");
      for (uint i = 0; i < _moduleinfo_array.length; i++)
      {
@@ -1070,9 +1073,26 @@
          debug(PRINTF) printf("\tmodule[%d] = '%.*s'\n", i, m.name);
          if (m.unitTest)
          {
-            (*m.unitTest)();
+            testCount++;
+            try {
+                (*m.unitTest)();
+            } catch (Exception e) {
+                printf("Failure in test for module '%.*s':\n", m.name);
+                printf("  %.*s on %.*s(%d): %.*s\n",
+                       e.classinfo.name, e.file, e.line, e.msg);
+                testFail++;
+            }
          }
      }
+    if (testOnly != 0 || testFail > 0)
+    {
+        printf("TESTS: %d  PASSED: %d  FAILED: %d\n",
+               testCount, testCount - testFail, testFail);
+        if (testFail == 0)
+            exit(0);
+        else
+            exit(1);
+    }
  }



These additions are fairly minor, and would make unit testing 
immeasurably better than it is right now. The sections I modified in 
Tango are mostly the same as the Phobos versions, so the patch should be 
almost exactly the same as mine above.

Comments?

  - Gregor Richards