Field notes on a failure I think every analytics team will meet. The economist Charles Goodhart made an observation about monetary policy, later phrased by the anthropologist Marilyn Strathern in the form most people quote: when a measure becomes a target, it ceases to be a good measure. I find it more useful in the plain form. A metric works as a measurement only while nobody is rewarded for moving it.

How a good metric goes bad

Take a support centre that tracks average handle time, the minutes an agent spends per call. It is a sensible number. Long calls cost money. Then a manager sets a target: under six minutes. Within weeks the average falls to 5.4. Is the centre more efficient?

Maybe. But look at what else an agent can do to hit six minutes: cut the call short, transfer difficult customers to another queue, promise a callback that never comes, close the ticket before the problem is fixed. Each of those moves the metric in the right direction and hurts the customer. The repeat-call rate rises a month later, and by then nobody connects it to the target.

Nobody has to be dishonest for this to happen. People respond to what is measured and rewarded, mostly without any conscious plan to cheat. That is why blaming individuals misses the point; the design of the measure is the problem.

Patterns I would look for

  • Gaming the definition. A delivery team measured on “orders delivered within 30 minutes” starts the clock when the rider picks up, not when the customer orders. The number improves, delivery does not.
  • Shifting the denominator. A sales team measured on conversion rate stops logging low-quality leads. Conversion rises, revenue is flat.
  • Timing games. Deals pulled into the last week of the quarter, or returns processed in the next month, so that the period’s number looks better.
  • Neglecting the unmeasured. If only on-time delivery is tracked, then damage in transit, rudeness and complaint handling quietly degrade.
  • Vanity growth. App installs, page views or “registered users” that are cheap to inflate with incentives and tell you little about use.

What actually helps

No single fix exists, and anyone who claims otherwise is selling something. The measures below reduce the damage.

Pair every target with a counter-metric. Handle time goes next to repeat-contact rate and customer satisfaction on the same dashboard, reviewed together. Gaming one then shows up in the other. In experiments these are called guardrail metrics, and the idea carries over directly to operations.

Prefer outcomes to activity. Calls made, tickets closed and emails sent are easy to count and easy to inflate. Problems resolved without a second contact is harder to compute and harder to fake.

Use ranges rather than thresholds. A target of “at least X” invites people to stop at X, or to cut corners to reach it. Reviewing a trend and a band, for example handle time between five and eight minutes depending on call type, takes the pressure off the single point.

Rotate or refresh metrics. A measure that has been a target for two years has been optimised for two years. Changing the emphasis now and then, and auditing a sample of the underlying cases by hand (listen to twenty calls, not just average the durations), keeps the measure honest.

Separate the people who define a metric from the people who are scored on it. When a team computes its own bonus number, the definition drifts in its favour. A neutral analytics or finance group owning the calculation is a cheap, effective control.

My opinion on dashboards generally

A dashboard is not neutral once leadership starts looking at it weekly. The act of displaying a number prominently signals that it matters. So choose what to display as if it will be gamed, because to some degree it will be. Ask of each tile: if someone wanted to improve this number without improving the business, how would they? If the answer comes quickly, add a counter-metric or change the tile.

And do not swing to the opposite error of refusing to set targets. Targets focus effort, and some metrics are hard to game. The aim is to hold the number lightly: look at it, but keep checking what it stands for.

Leave a Reply

Trending