What does Coverity/clang static analysis actually do?

[Brad Roberts]

Walter Bright wrote:
> Brad Roberts wrote:
>>>> * graphing of issue trends
>>> That's a crock <g>.
>>
>> Uh, whatever.  Most of the rest of us humans respond much better to
>> pictures and trends than to raw numbers.  Show me some visual
>> indication of the quality of my code (ignoring the arguments about
>> the validity of such graphs) and I can pretty much guarantee that
>> I'll work to improve that measure.  Nearly everyone I've ever worked
>> with behaves similarly.. once they agree that the statistic being
>> measured is useful.  One of the best examples is percent of code
>> covered by unit tests.  The same applies to number of non-false
>> positive issues discovered through static analysis.
>>
> 
> 
> A long time ago, the company I worked for decided to put up a huge chart
> on the wall that everyone could see, and every day the current bug count
> was plotted on it. The idea was to show a downward trend.
> 
> It wasn't very long (a few days) before this scheme completely backfired:
> 
> 1. engineers stopped submitting new bug reports
> 
> 2. the engineers and QA would argue about what was a bug and what wasn't
> 
> 3. multiple bugs would get combined into one bug report so it only
> counted once
> 
> 4. if a bug was "X is not implemented", then when X was implemented,
> there might be 3 or 4 bugs against X. Therefore, X did not get implemented.
> 
> 5. there was a great rush to submit half-assed fixes before the daily
> count was made
> 
> 6. people would invent bugs for which they would simultaneously submit
> fixes (look ma, I fixed all these bugs!)
> 
> 7. arguing about it started to consume a large fraction of the
> engineering day, including the managers who were always called in to
> resolve the disputes
> 
> In other words, everyone figured out they were being judged on the
> graph, not the quality of the product, and quickly changed their
> behavior to "work the graph" rather than the quality.
> 
> To the chagrin of the QA staff, management finally tore down the chart.
> 
> Note that nobody involved in this was a moron. They all knew exactly
> what was happening, it was simply irresistible.

Existence of a bad case doesn't disprove the usefulness in general.  Yes, I
agree that number of bugs is a bad metric to measure all by itself.

Water can drown a person, but that doesn't make it something to avoid.

Sigh,
Brad