Hack to name unit tests?

[Robert Fraser]

Dan wrote:
> Christopher Wright Wrote:
> 
>> Robert Fraser wrote:
>>> Christopher Wright wrote:
>>>> Robert Fraser wrote:
>>>>> I'm creating a Flectioned-based way to run unit tests individually 
>>>>> (it'll be hooked into a front-end, so you can run test suites, repeat 
>>>>> only failed tests after a code change, etc.)
>>>>>
>>>>> However, I have run into a problem: unit tests can't be named. Right 
>>>>> now I have a signature system, but I was wondering if there's any 
>>>>> hackish way to predictably insert a string into the generated object 
>>>>> file in a way that users can name unittests. That is to say, given a 
>>>>> function pointer, is there any way I can scan the code to see if a 
>>>>> name has been given to the unit test, and if so extract that name at 
>>>>> runtime?
>>>>>
>>>>> Ideally, it'd be as easy for the user as something like:
>>>>>
>>>>> unittest
>>>>> {
>>>>>    assert(":testName:");
>>>>>    // Rest of test goes here...
>>>>> }
>>>>>
>>>>> Right now, I have the associations being done in the front-end I'm 
>>>>> working on, that scans the code for comments in a particular format 
>>>>> and associates those with the unit tests. However, that ties the unit 
>>>>> test executor back-end to the code analysis front-end, so I was 
>>>>> hoping there's a better way.
>>>>>
>>>>> Thanks!
>>>> class Unittest (string _name, alias _dg) : IUnittest {
>>>>    static string name = _name;
>>>>    static void delegate() test = _dg;
>>>> }
>>>>
>>>> Usage:
>>>> Unittest!("my test for wossname", {
>>>>    assert (false, "haven't gotten around to implementing this yet");
>>>> });
>>>>
>>>> Not guaranteed to work.
>>>>
>>>> Another tactic would be something like:
>>>>
>>>> class UnittestManager {
>>>>    static void registerCurrentTest (string name) {}
>>>>    static void endTest (bool success) {}
>>>>    static bool performTest () {}
>>>> }
>>>>
>>>> template Unittest (string _name, alias _dg) {
>>>>    unittest {
>>>>       UnittestManager.registerCurrentTest(_name);
>>>>
>>>>       // This lets us skip the test if we're just trying
>>>>       // to find out what tests there are currently.
>>>>       if (UnittestManager.performTest) {
>>>>          scope(success) UnittestManager.endTest(true);
>>>>          scope(failure) UnittestManager.endTest(false);
>>>>          _dg();
>>>>       }
>>>>    }
>>>> }
>>>>
>>>> mixin Unittest!("my test name", { assert (false, "not yet 
>>>> implemented"); });
>>> Thanks! That's a good idea, but I want something compatible with current 
>>> unittest {} declarations.
>> You already have to modify the tests to insert the name, but you want 
>> something that works as a decorator inside the unittest. Can your 
>> unittest runner do a try/catch on a unittest? Then you could have 
>> something like:
>>
>> void testName(string name) {
>>     if (collectingNames)
>>        throw new UnittestNameException(name);
>> }
>>
>> struct NamedTest {
>>     Unittest test;
>>     string name;
>> }
>>
>> ...
>> void main () {
>>     NamedTest[] tests;
>>     collectingNames = true;
>>     foreach (test; getUnittests) {
>>         try {
>>             test();
>>         } catch (UnittestNameException e) {
>>             tests ~= NamedTest(test, e.name);
>>         }
>>     }
>>     collectingNames = false;
>> }
>>
>> unittest {
>>     testName("blargh");
>>     assert (false, "still working on this");
>> }
> 
> Eww... no.  You guys are suggesting that code provability is useless in D because it's "too complicated" and then go on to suggest it can be used as a heavily over-classed debug statement.
> 
> unittest is currently used more or less as a place to put a barrage of random sample tests.  I was suggesting (beyond the obvious improvement of naming and only executing select unittests) that we should be able to put hooks to a provability program here.
> 
> The provability program would be a separate utility that would go through and show things like bit overflow potential in numbers, potential infinities or potential bad memory accesses.  It would do these things by following variables throughout their lifetimes to show all the different values they could possibly be at each time they're used.
> 
> Combined with contract programming (in/out and invariant) this kind of thing exposes code in a way that running it with test values simply can't.
> 
> For example:
> 
> In libc, in qsort(), there was a bug because:
> 
> median = (low + high) >> 1;
> 
> The problem here is that in a very large array, low + high will eventually be > MAX_INT.  This kind of bug isn't detected by simply doing a dumb random test on an array of 10.  You need to be aware of the full range of possible values and use inductive logic to predict the bug without testing every last case.
> 
> Regards,
> Dan

It's not a bad idea, especially for smaller algorithms, but from my 
experience, using Coverity on a ~18 million LOC Java project fixed about 
10 bugs, while the unit tests were constantly auditing check-ins, and 
prevented me from checking in at least that many of my own.