Suggestion (ping Walter): Improve unit testing.

[Tom]

I agree. Unit testing needs a lot of improvement to be really useful.

--
Tom;
(Tomás Rossi)

Gregor Richards escribió:
> There are a number of improvements I'd like to see in D's unit testing 
> framework. All of them are standard library changes, and I've made them 
> work in Tango (as an external patch). If they were added to both Phobos 
> and Tango, DSSS could easily run unit tests, which would improve its 
> test suite.
> 
> Here are two major problems I see with unit tests right now:
> 1) You can't run only unit tests, you are forced to run main() as well.
> 2) Unit testing doesn't actually output anything useful, it just works 
> under the no-news-is-good-news principle.
> 
> Each problem has a fairly easy solution in the core library:
> 
> 
> 
> 1) Adding a special option, handled by the core library's main function, 
> which will cause only unit tests to run. The changes I made in Tango:
> Index: lib/compiler/gdc/dgccmain2.d
> ===================================================================
> --- lib/compiler/gdc/dgccmain2.d    (revision 2100)
> +++ lib/compiler/gdc/dgccmain2.d    (working copy)
> @@ -35,7 +35,7 @@
>  extern (C) void _minit();
>  extern (C) void _moduleCtor();
>  extern (C) void _moduleDtor();
> -extern (C) void _moduleUnitTests();
> +extern (C) void _moduleUnitTests(int);
> 
>  /***********************************
>   * These functions must be defined for any D program linked
> @@ -108,6 +108,7 @@
>  {
>      char[][] args;
>      int result;
> +    int testOnly = 0;
> 
>      version (GC_Use_Stack_Guess)
>      {
> @@ -137,11 +138,17 @@
>          }
>          args = am[0 .. argc];
>      }
> +
> +    // check for --d-unittests-only
> +    foreach (arg; args) {
> +        if (arg == "--d-unittests-only")
> +            testOnly = 1;
> +    }
> 
>      void run()
>      {
>          _moduleCtor();
> -        _moduleUnitTests();
> +        _moduleUnitTests(testOnly);
>          result = main_func(args);
>          isHalting = true;
>          _moduleDtor();
> 
> 
> As you can see, if --d-unittests-only is supplied, it passes a 1 into 
> the (now modified) _moduleUnitTests function, which will cause it to 
> exit when finished testing.
> 
> 
> 
> 2) The _moduleUnitTests function can be improved to provide useful 
> output. The output I've implemented for Tango is:
> Failure in test for module 'test':
>   tango.core.Exception.AssertException on test.d(7): Assertion failure
> TESTS: 1  PASSED: 0  FAILED: 1
> 
> The patch to make this work was fairly simple as well:
> Index: lib/compiler/gdc/genobj.d
> ===================================================================
> --- lib/compiler/gdc/genobj.d    (revision 2100)
> +++ lib/compiler/gdc/genobj.d    (working copy)
> @@ -42,7 +42,8 @@
>      import tango.stdc.string; // : memcmp, memcpy;
>      import tango.stdc.stdlib; // : calloc, realloc, free;
>      import util.string;
> -    debug(PRINTF) import tango.stdc.stdio; // : printf;
> +    //debug(PRINTF) import tango.stdc.stdio; // : printf;
> +    extern (C) int printf(char*, ...);
> 
>      extern (C) void onOutOfMemoryError();
>      extern (C) Object _d_newclass(ClassInfo ci);
> @@ -1054,11 +1055,13 @@
>  }
> 
>  /**
> - * Run unit tests.
> + * Run unit tests. If testOnly is true, output results and quit.
>   */
> 
> -extern (C) void _moduleUnitTests()
> +extern (C) void _moduleUnitTests(int testOnly)
>  {
> +    int testCount, testFail;
> +    testCount = testFail = 0;
>      debug(PRINTF) printf("_moduleUnitTests()\n");
>      for (uint i = 0; i < _moduleinfo_array.length; i++)
>      {
> @@ -1070,9 +1073,26 @@
>          debug(PRINTF) printf("\tmodule[%d] = '%.*s'\n", i, m.name);
>          if (m.unitTest)
>          {
> -            (*m.unitTest)();
> +            testCount++;
> +            try {
> +                (*m.unitTest)();
> +            } catch (Exception e) {
> +                printf("Failure in test for module '%.*s':\n", m.name);
> +                printf("  %.*s on %.*s(%d): %.*s\n",
> +                       e.classinfo.name, e.file, e.line, e.msg);
> +                testFail++;
> +            }
>          }
>      }
> +    if (testOnly != 0 || testFail > 0)
> +    {
> +        printf("TESTS: %d  PASSED: %d  FAILED: %d\n",
> +               testCount, testCount - testFail, testFail);
> +        if (testFail == 0)
> +            exit(0);
> +        else
> +            exit(1);
> +    }
>  }
> 
> 
> 
> These additions are fairly minor, and would make unit testing 
> immeasurably better than it is right now. The sections I modified in 
> Tango are mostly the same as the Phobos versions, so the patch should be 
> almost exactly the same as mine above.
> 
> Comments?
> 
>  - Gregor Richards