Program logic bugs vs input/environmental errors

[Dicebot via Digitalmars-d]

On Thursday, 16 October 2014 at 20:18:04 UTC, Walter Bright wrote:
> On 10/16/2014 12:56 PM, Dicebot wrote:
>> On Thursday, 16 October 2014 at 19:35:40 UTC, Walter Bright 
>> wrote:
>>> Ok, but why would 3rd party library unittests be a concern? 
>>> They shouldn't
>>> have shipped it if their own unittests fail - that's the 
>>> whole point of having
>>> unittests.
>>
>> Libraries tend to be forked and modified.
>
> If you're willing to go that far, then yes, you do wind up 
> owning the unittests, in which case s/assert/myassert/ should 
> do it.

Which means changing almost all sources and resolving conflicts 
upon each merge. Forking library for few tweaks is not "going 
that far", it is an absolutely minor routine. It also complicates 
propagating changes back upstream because all tests need to be 
re-adjusted back to original style.

>> Libraries aren't always tested in
>> environment similar to specific production case.
>
> Unittests should not be testing their environment. They should 
> be testing the function's logic, and should mock up input for 
> them as required.

compiler version, libc version, kernel version - all it can 
affect behaviour or pretty self-contained function. Perfectly 
tested library is as much reality as program with 0 bugs.

>> At the same time not being able
>> to use same test runner in all Continious Integration jobs 
>> greatly reduces the
>> value of having standard unittest blocks in the very first 
>> place.
>
> I understand that, but wouldn't you be modifying the unittests 
> anyway if using an external test runner tool?

No, right now one can affect the way tests are run by simply 
replacing the runner to a custom one and it will work for any 
amount of modules compiled in. Beauty of `unittest` block 
approach is that it is simply a bunch of functions that are 
somewhat easy to discover from the combined sources of the 
program - custom runner can do pretty much anything with those. 
Or it could if not the issue with AssertError and cleanup.