The belief: a model with 99 percent accuracy is excellent. It sounds like an exam score, and anything above 90 feels like a pass. Let us take it apart with a deliberately lazy model.
A lender has 10,000 customers this month, and 100 of them will stop paying their instalments. That is 1 percent. Now build a “model” that predicts, for every customer, that they will pay. It requires no data and no training. Its accuracy is 9,900 correct out of 10,000, which is 99 percent. It also catches zero defaulters, which was the only reason for building a model.
The flaw is that accuracy counts all correct calls equally, and when one class is rare, the common class swamps everything. Whenever you are looking for something uncommon, such as fraud, churn, defaults, faulty parts or serious illness, accuracy is the wrong headline number.
Two questions instead of one
Switch to a more useful model. It flags 150 customers as likely defaulters. When the month ends, 60 of those flagged did default, and 90 did not. The other 40 defaulters were not flagged.
Precision asks: of those flagged, how many were truly defaulters? That is 60 of 150, or 40 percent. Recall (also called sensitivity) asks: of all the real defaulters, how many did we catch? That is 60 of 100, or 60 percent. Both are fractions with the same numerator, true positives, but different denominators: what we flagged versus what really existed.
Its accuracy, by the way, is (60 correct flags + 9,810 correct non-flags) out of 10,000, which is 98.7 percent. Lower than the lazy model’s 99 percent, and far more useful. That alone should end the argument.
The trade-off you cannot escape
Most models output a score, and you pick a cut-off. Lower the threshold and you flag more customers, which catches more real defaulters (recall rises) but also sweeps in more innocent ones (precision falls). Raise it and the reverse happens. There is no setting that maximises both, so the question is which error costs more.
- A collections team can make only 80 calls a day. Wasted calls are expensive, so precision matters.
- A screening step that sends borderline cases to a human reviewer is cheap per case, while a missed case is costly. Recall matters more.
- A spam filter that hides a real offer letter in the junk folder is worse than letting one spam message through. Precision on the spam class matters.
Put rupees on it. If chasing a flagged customer costs ₹300 in staff time and a caught default saves ₹20,000, then 60 catches yield ₹12,00,000 for 150 calls costing ₹45,000. Cheap calls and a large prize push you to lower the threshold, even at poor precision. If each call cost ₹5,000, the same arithmetic would tell you to be pickier. The metric follows the economics, not the other way round.
F1: a compromise with a hidden opinion
The F1 score is the harmonic mean of precision and recall: twice their product over their sum. For our model, 2 x 0.4 x 0.6 divided by (0.4 + 0.6) is 0.48. The harmonic mean punishes imbalance. A model with precision 1.0 and recall 0.02 has an arithmetic mean over 0.5 but an F1 near 0.04, which is the honest verdict.
F1 treats precision and recall as equally important, and in a real business they rarely are. If you care more about recall, the generalisation F-beta lets you weight it. More often, the better practice is to skip the single number and report precision and recall at the threshold you actually plan to use, along with the cost per outcome.
Smaller points that bite
A baseline matters. Before celebrating 40 percent precision, note that picking customers at random would give 1 percent, since that is the base rate. So the model is forty times better than chance. Quote that ratio, often called lift, next to the raw figure.
Precision depends on how common the target is. The same model applied to a portfolio where 5 percent default would show much higher precision, with no change in the model. Compare precision across datasets only if the base rates are comparable.
Finally, evaluate on data the model has not seen, and ideally from a later period. A model graded on its own training data will look better than it is. Then the numbers above become something you can plan staffing around, rather than a slide.
Figures and names in the examples are illustrative.
Leave a Reply