Documented unittests & code coverage

[Jonathan M Davis via Digitalmars-d]

On Thursday, July 28, 2016 16:14:42 Walter Bright via Digitalmars-d wrote:
> On 7/28/2016 3:15 AM, Johannes Pfau wrote:
> > And as a philosophical question: Is code coverage in unittests even a
> > meaningful measurement?
>
> Yes. I've read all the arguments against code coverage testing. But in my
> usage of it for 30 years, it has been a dramatic and unqualified success in
> improving the reliability of shipping code.

The issue isn't whether we should have code coverage testing. We agree that
that's a great thing. The issue is whether the lines in the unit tests
themselves should count towards the coverage results.

https://issues.dlang.org/show_bug.cgi?id=14856

gives some good examples of why having the unittest blocks themselves
counted in the total percentage is problematic and can lead to dmd's code
coverage tool listing than 100% coverage in a module that is fully tested.
What's critical is that the code itself has the coverage testing not that
the lines in the tests which are doing that testing be counted as part of
the code that is or isn't covered.

I know that it will frequently be the case that I will not get 100% code
coverage per -cov for the code that I write simply because I frequently do
stuff like use scope(failure) writefln(...) to print useful information on
failure in unittest blocks so that I can debug what happened when things go
wrong (including when someone reports failures on their machine that don't
happen on mine).

D's code coverage tools are fantastic to have, but they do need a few tweaks
if we want to actually be reporting 100% code coverage for fully tested
modules. A couple of other reports that I opened a while back are

https://issues.dlang.org/show_bug.cgi?id=14855
https://issues.dlang.org/show_bug.cgi?id=14857

- Jonathan M Davis