Unit tests called multiple times?

[Steven Schveighoffer]

On Fri, 12 Apr 2013 19:16:05 -0400, H. S. Teoh <hsteoh at quickfur.ath.cx>  
wrote:

> On Sat, Apr 13, 2013 at 01:08:38AM +0200, William wrote:
>> On Friday, 12 April 2013 at 23:01:55 UTC, Steven Schveighoffer
>> wrote:
>> >On Fri, 12 Apr 2013 18:52:50 -0400, William
>> ><squidkidsignup at gmail.com> wrote:
>> >
>> >>I'm compiling a medium-sized project written in D with the
>> >>-unittests flag.  I have some writeln statements in my unit
>> >>tests, for debug purposes, and I've noticed that certain unit
>> >>tests are being run more than once.  Sample output is along
>> >>these lines:
>> >>
>> >>DEBUG: unit test started...
>> >>DEBUG: loop found whatever
>> >>DEBUG: iteration successful
>> >>DEBUG: bar is equal to bar
>> >>DEBUG: unit test passed...
>> >>DEBUG: unit test started...
>> >>DEBUG: loop found whatever
>> >>DEBUG: iteration successful
>> >>DEBUG: bar is equal to bar
>> >>DEBUG: unit test passed...
>> >>DEBUG: unit test started...
>> >>DEBUG: loop found whatever
>> >>DEBUG: iteration successful
>> >>DEBUG: bar is equal to bar
>> >>DEBUG: unit test passed...
>> >>
>> >>is this standard behavior for DMD?  Is it documented anywhere?
>> >> It's not much of a problem, but it strikes me as rather odd,
>> >>and makes the debug output difficult to read.
>> >
>> >Are your unit tests inside templates?  If so, they are
>> >instantiated once per template.  To fix, move outside the
>> >template.
>> >
>> >This actually can be a nice feature, and at the same time
>> >horrible.  It's nice because you can parameterize your unit tests!
>> >It's horrible because you have to be careful your unit tests are
>> >valid for all possible instantiations!
>> >
>> >-Steve
>>
>> Thank you!  That was exactly it.  I tend to put my unittests directly
>> after the functions/classes they test, so my opApply for an
>> implementation of n-dimensional associative arrays, (with
>> parameterized keys/values) was having its associated unittest called
>> every time the template was instantiated.  Moving it to the module
>> level fixed the problem, although now it reads as though the test is
>> intended to test the whole class, when in fact it only verifies the
>> correctness of opApply.  Still, feels good to have weird behavior
>> demystified.
>
> You could use static if to make the unittest only instantiate for a
> particular template instance, e.g.:
>
> 	struct S(T) {
> 		...
> 		void method(T t) { ... }
>
> 		static if (T==int)
> 		unittest {
> 			auto s = S!int(123);
> 			s.method(123);
> 			assert( ... );
> 		}
> 	}
>
> This way, you can still keep unittests together with the methods they
> test. Beware, though, that you actually instantiate S!int somewhere
> outside, otherwise the unittest may never actually be instantiated and
> run. :)

This is similar to what dcollections does:

https://github.com/schveiguy/dcollections/blob/master/dcollections/ArrayList.d#L53

https://github.com/schveiguy/dcollections/blob/master/dcollections/ArrayList.d#L146

> (This does give rise to an interesting idiom of unittests catered for
> specific types that only run when your code actually uses those
> particular instantiations. Hmm...)

The issue mainly is that unit tests generally provide the parameters, so  
if the unit test is parameterized, you have to parameterize your  
literals!  It's not very easy.  For example, see how I have to use casts  
in the ArrayList unit test above.  I also have helper functions in some  
cases that allow me to construct the correct type.

-Steve