SQLite testing procedures

[Robert Clipsham]

On 05/08/2011 21:22, Jonathan M Davis wrote:
> On Friday 05 August 2011 21:13:17 Robert Clipsham wrote:
>> This is a fairly old document now, but it has some interesting ideas
>> that could have a place in D (either the language, tools, or phobos):
>>
>> http://www.sqlite.org/testing.html
>>
>> Some notable things other than the range of testing methods used:
>>    * 100% branch test coverage (7.1)
>>       - It would be nice if dmd's -cov offered a way to give the coverage
>> for statements and branches rather than just per line
>
> But each branch of code is on a particular branch, so if every branch is hit,
> every line is hit. Do you mean cases such as where you have 5 if statements in
> a row, and every one of them gets hit at one time or another, but in some
> cases the execution path goes through 3 of them or 2 of them or whatever, but
> not every single combination is tested? That would be difficult to do given that
> there's a good chance that certain combinations of branches should never
> actually occur. It would also complicate the implementation of -cov
> considerably I would think.

// This gets 100% coverage currently if a == 1. No testing of the
// second branch is necessary. Or vice-versa.
if (a == 1 || b > 2)
{
}

With branch based coverage, both branches must get tested for 100% 
coverage - SQLite's testing is incredibly thorough, and it shows in 
their rock solid product.

>
>>    * Coverage testing of defensive code (7.2)
>>      - Some branches are always (or never) executed, which can mean 100%
>> branch coverage is not possible unless you remove the "defensive" code.
>>      - SQLite solves this with the ALWAYS() and NEVER() macros
>>      - This would only become relevant if -cov was improved
>
> It already happens. If you use assert(0) in your code anywhere (prime examples
> of that being in the default case of a switch statement which should never
> happen or in a catch block which should never be hit but is required to make
> the function nothrow because a function in the try block theoretically _could_
> throw but never actually will with the arguments that it's given in that
> case). std.datetime has very high code coverage (100% aside from the assert(0)
> lines, I believe), but the assert(0) lines are naturally never hit (it would
> be a bug if they were), so -cov does not actually give it 100% (it gets
> something more like 98%).

It might be worth reading the article about this, I can't summarize it 
sufficiently.

>
> - Jonathan M Davis


-- 
Robert
http://octarineparrot.com/