Quit running foreign unittests >_<

[Steven Schveighoffer via Digitalmars-d]

On 4/28/15 12:40 PM, Dicebot wrote:
> On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote:
>> On 4/27/15 6:20 AM, Dicebot wrote:
>>> On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:
>>>> On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
>>>>> Compiling tests of dependencies pretty much never causes any notable
>>>>> slowdown.
>>>>
>>>> This thread doesn't support that view, see the first post.
>>>
>>> Which part exactly? I only see comparisons for compiling AND running
>>> tests for dependencies. And it is usually running which causes the
>>> slowdown.
>>
>> The problem is as follows:
>>
>> 1. Unit tests for some library are written for that library. They are
>> written to run tests during unit tests of that library only (possibly
>> with certain requirements of environment, including build lines, or
>> expectations of system resource availability).
>> 2. People who import that library's modules are not trying to test the
>> library, they are trying to test their code.
>
> Those are two points I fundamentally disagree with.

I think by default, nobody wants to test already-tested code in their 
application. That's not the point of unit tests.

For example, if there's a module that has:

struct S(T)
{
   unittest {...}
}

unittest
{
    S!int; // run unit tests for int
}

Then I can assume that unit test was run and passed. If I want to test 
it to be sure, I'll run that library's unit tests!

But I don't want my unit test that uses S!int to also test S's unit 
tests for S!int. That doesn't make sense. I'm wasting cycles on 
something that is already proven.

Now, if I want to run unit tests for S!MyCustomType, that makes sense. 
The library didn't test that. Which is why there should be a way to do 
it (if it's valid!).

Right now, there isn't a way to control what runs and what doesn't. I 
don't care where it's decided or how it's decided, but somehow I should 
be able to NOT run S!int tests again.

> Unless compiling some specific tests causes some proven _compilation_
> slowdown (I have yet to see that) those all must be compiled and
> filtered by runtime test runner optionally.

For example, RedBlackTree when running unit tests does a sanity check of 
the tree for every operation. If you then use a RedBlackTree in your 
unit tests, you are doing that again. Maybe it's worth it to you, but it 
can increase the runtime of your test by orders of magnitude. Last time 
I checked, performance was high on people's priority lists.

Or, potentially you could be running INVALID TESTS, because the tests 
weren't written for your specific usages. I ran into this when others 
complained that they couldn't test their code that uses 
RedBlackTree!string, because all the unit tests instantiated with int 
literals. This is simply a "does not work" thing. You can't turn them 
off, and you can't test your code. Is it worth it to you for a library 
to try to compile tests that prevent your test build from happening? Do 
you enjoy waiting on others to fix their problems so you can test your 
code? This is a very less than ideal situation. And it's not something 
the compiler tells you will happen.

> And if tests are written in a weird way that they can only be ran within
> that library test step, those are not really unittests.

This means you should never ever write unit tests into a template. 
Because it's impossible to create tests that will work for every single 
instantiation. See RedBlackTree for all the crud you have to put in for 
this to be viable today. I'd love to get rid of all that.

In which case, let's disallow that capability. I'm fine with that too. 
This simply means your "all encompassing tests" will not be all 
encompassing any more, they will only test a select few instantiations, 
and you can't control that either from your application code.

I want a way to control it as a template designer and as a template 
instantiator. Flexibility is king.

-Steve