The economist Charles Goodhart observed in the 1970s that a statistical regularity tends to collapse once it is used for control. The anthropologist Marilyn Strathern later put it in the form most people quote: when a measure becomes a target, it ceases to be a good measure. Quality engineering is full of examples. Nearly every common QA metric is a reasonable signal when it is observed, and a misleading one when people are rewarded for moving it.

This matters because QA metrics are often the main way leadership sees quality. If the numbers can be improved without the product improving, leadership ends up confident about the wrong things.

Bugs found per tester

Counting defects raised by each tester looks like a fair productivity measure. Make it a target and three things happen. One root cause gets logged as five bugs, one per screen where it shows up. Cosmetic issues are logged eagerly because they are quick, while hard investigations into intermittent failures are avoided because they may produce nothing for days. And developers, seeing their modules attract large numbers, start disputing every ticket, which damages the working relationship the count was supposed to reflect.

There is also a basic logical problem. The number of bugs found depends heavily on how many bugs exist. A tester assigned to a well-built module will find fewer than one assigned to a rushed one. Rewarding the count rewards being assigned to bad code.

Test case pass rate

“98% of test cases passed” is a familiar line in release reports. As a target, it is easy to game: write many trivial cases that always pass, mark failing cases as blocked or not applicable, or quietly retire tests that keep failing. Even without gaming, the percentage hides what matters. If 490 of 500 cases pass, the 10 failures could be cosmetic, or one of them could be “payment debited but order not created”. The rate is identical.

A more honest report lists the failing cases by risk, not by count. A release meeting needs to hear “two open issues affect checkout, one affects the invoice PDF”, not a percentage.

Code coverage

Coverage measures which lines or branches executed during tests. Low coverage is genuinely informative: code that never runs in any test is certainly untested. High coverage says much less, because a test can execute a line without checking its result. Set a mandatory 85% coverage gate and some teams will write tests that call functions and assert nothing, or assert only that no exception was thrown. The coverage report turns green; the bug-catching ability barely changes.

Mutation testing is one way to check whether tests actually verify behaviour: a tool such as PIT for Java or Stryker for JavaScript makes small changes to the code (flipping a comparison, changing a constant) and checks whether any test fails. Surviving mutants reveal code that is covered but not really tested. It is slower and noisier than coverage, but it measures something closer to what people hope coverage measures.

Number of automated tests

“We automated 1,200 test cases this year” sounds like progress. As a target it encourages splitting one meaningful test into several thin ones, automating easy low-value cases first, and never deleting obsolete tests. A suite that grows without pruning becomes slower and flakier, which reduces the value of every test in it. What leadership actually wants to know is how quickly the team gets trustworthy feedback on a change and how much manual effort remains before release.

Defects leaked to production

Escaped defects are one of the more meaningful metrics, because they measure outcomes rather than activity. Even this one bends under pressure. If QA is blamed for every escape, teams argue about classification (“that was a requirements gap, not a test miss”), production issues get logged as change requests, and support teams are encouraged to resolve tickets without creating defects. The number improves while customers see no difference.

The signal survives better when escapes are treated as information about the whole delivery system, analysed without blame, and paired with severity and customer impact rather than reported as a raw count.

Patterns that keep metrics useful

Metrics are not the problem. Using single numbers as targets, especially for individuals, is. A few practices reduce the damage:

  • Pair metrics that pull against each other. Release frequency with change failure rate. Defects found before release with defects found after. Automation count with suite run time and flaky rate. Gaming one usually shows up in its partner.
  • Prefer outcome metrics to activity metrics. Customer-reported incidents, time to restore service and time from commit to production say more than cases executed or bugs logged.
  • Look at trends, not thresholds. A coverage figure drifting down over six months is a useful conversation; a hard gate at a fixed number invites workarounds.
  • Keep individual performance separate. Team metrics describe a system. Using them to rank individuals almost guarantees distortion. Judge individuals through the quality of their work, peer feedback and the problems they prevent, which requires a manager who actually looks.
  • Read the numbers with the people who produce them. A metric reviewed with the team (“why did escapes rise this quarter?”) stays a diagnostic tool. A metric reported upwards without context becomes a target.

A quick self-check

For any metric on a QA dashboard, ask three questions. Could someone improve this number without the product getting better? Would a team doing excellent work ever see this number get worse? Does anyone’s appraisal depend on it? If the answers are yes, yes and yes, the metric is probably already misleading you, and the fix is to change how it is used, not to find a cleverer formula.

A good QA metric is one that prompts a useful question. The moment it starts producing defensive answers instead, it has crossed Goodhart’s line.

Written by the RP Strativa Team. Figures in the examples are illustrative.

Leave a Reply

Trending