Improving unit tests

[Andrei Alexandrescu]

Bill Baxter wrote:
> On Sat, Nov 8, 2008 at 9:48 AM, Derek Parnell <derek at psych.ward> wrote:
>> On Fri, 07 Nov 2008 08:25:04 -0800, Janderson wrote:
>>
>>> I'd like to learn about move advanced features that D is missing.
>> The D language does *not* do unit testing. At best, it provides a place in
>> which the coder can put their unit testing code and an automated initiation
>> of the coder's unit test code.
>>
>>
>>> I was originally thinking, maybe unit tests shouldn't be part of D to
>>> allow for innovation. However then I though, what about if D's unit
>>> tests where extensible though some language syntax?
>>>
>>> Questions:
>>> 1) What features are missing from D's unit tests that you miss?
>> ** A variation of the assert statement that does not halt execution upon
>> finding an error. Instead, it just reports the error and moves on to the
>> next test.
> 
> Can't that be done by a library function?  I've seen in Python
> unittesting the use of "assert_" instead of the built-in "assert".  I
> assume it's for the same reason.
> 
>> ** A method of identifying the failing test other than by file-name and
>> line-number.
> 
> Seems like a lib could do this too.
> Just some sort of   BeginTests("Module identification") that's tied
> into the definition of that assert_()
> 
>> ** A method by by the number of failures and successes are recorded and
>> reported at the end of execution. Possibly event made availble in symbolic
>> form to the coder for whatever purposes they choose. eg. Keeping a database
>> of unit ttest results over time.
> 
> Can that be done with a lib?

I agree with the spirit that we need to look more into what we can do 
with libraries. At this stage in D's development, this group is a 
crucible of new language features, which is good, but oughtn't deter us 
from looking at libraries.

>> In addition to those above...
>> ** Being able to run unit tests on a per-module basis without the need of a
>> main() function.
> 
> Yeh, that one is annoying.  You shouldn't need to have a main to run
> unit tests.  If there isn't one the compiler should supply an empty
> one when unittesting.

At a point I'd added --main to rdmd. It defines a vacuous void main(){} 
to the banana. (I removed it since because I thought it's too minor a 
feature.) When I talked to Walter about that, he said he'd consider 
adding such a switch to dmd.

>> ** Being able to import and declare symbols so that they are only in scope
>> for their unit test block.
> 
> Agreed.  version(UnitTests) { import module_for_tests; unittest { .. } }
 >
> Is annoying.  Being able to stick that import inside the unittest{ ...
> } would be much nicer.

For a good time, try version(unittest). Yours truly's brainchild :o).

>> ** Nesting of unit test blocks.
> 
> You mean nesting unittest{  unittest{ .. } ..} What would that be for?
> 
> But on the subject of nesting, I've noticed it's not possible to put
> unittests _inside_ classes and structs.  Or inside functions to test
> nested functions.  I guess there is a problem of supplying an
> appropriate context there.  Not sure if it can be solved.  For classes
> I think it would be ok if you just treat unittest{} like a static
> method.

I agree these should be added to the language, and indeed as static 
blocks. Moreover, unittest should in fact be put inside functions too 
and treated as a "run-once" block. But maybe that goes a bit too far :o).


Andrei