Pull up a chair. You have been handed a fraud-detection model to “test”, and you are wondering where to begin, because there are no screens and no requirements document, only a data scientist saying “accuracy is 99 percent”. Let us go through what you would actually do, and why that 99 should make you curious instead of comfortable.
Why 99 percent can mean nothing
Imagine 10,000 card transactions, of which 100 are fraud. A lazy model that says “not fraud” for everything is right 9,900 times. That is 99 percent accuracy, and it catches no fraud at all. So with a rare event, accuracy is the wrong lens.
Two numbers do better. Recall asks: of the real frauds, how many did we catch? Precision asks: of the transactions we flagged, how many were really fraud? Suppose the real model flags 150 transactions and 60 are genuine fraud. Precision is 60 / 150 = 0.40 and recall is 60 / 100 = 0.60. Now the conversation is useful: forty percent of flagged customers are innocent people whose card gets blocked, and forty percent of fraud still slips through. Is that acceptable? It is a business question, and your job is to put it on the table in numbers.
The model usually outputs a score, and a threshold turns the score into a decision. Move the threshold up and precision rises while recall falls; move it down and the reverse happens. Testing includes checking what the chosen threshold does, and who agreed to it.
Look at slices, not just the average
An overall score is an average, and averages hide groups. Break the results down by things that matter: city tier, new versus old customers, UPI versus card, device type, language, time of day, transaction amount band. Imagine recall is 0.60 overall but only 0.25 for transactions from customers who joined in the last month, because the training data had few of them. The headline looks healthy while one group is nearly unprotected, or, in a lending model, unfairly refused. Slice-level checks are the single most productive habit in model testing. Ask the team which slices the business cares about, and add the ones they have not thought of.
Check the data before you check the model
Much of what goes wrong is in the data, so a tester can contribute well here without knowing any mathematics. Look for:
- Leakage: a column in training that would not be available at prediction time, such as “chargeback_filed”. The model looks brilliant offline and fails live.
- Test data that overlaps with training data, which inflates every score.
- Missing values, impossible values (age 240, negative amounts) and units that differ between sources, such as rupees against paise.
- A training set from one period only: festival season, or the quiet months, but not both.
The same checks apply to the live data feeding the model. A pipeline that quietly starts sending a field as text instead of a number can degrade predictions without a single error being logged.
Behavioural tests, which you can write like normal tests
You can still write ordinary tests around a model, based on properties it ought to have:
- Invariance: changing something irrelevant should not change the answer. Same transaction with a different customer name spelling should score nearly the same.
- Direction: raising a loan applicant’s income while holding everything else fixed should not lower their approval score.
- Edge inputs: empty fields, extreme amounts, an unseen category. The service should return something sensible or a clear error, not crash.
- Known hard cases: a saved set of past examples the team already got wrong, so the same mistake cannot return unnoticed.
Do not forget the plain software around the model: response time under load, behaviour when the model service is down (does the payment go through unscored, or fail?), version tracking so you know which model made which decision, and logging that lets you explain a specific decision later.
After release: drift
A model is a photograph of the past, and the world keeps moving. Two kinds of change are worth knowing by name. Data drift means the inputs shift, such as a new payment app suddenly accounting for a third of traffic. Concept drift means the relationship between inputs and outcomes changes, such as fraudsters altering their tactics, so patterns that once signalled fraud no longer do. Neither throws an exception. Accuracy simply erodes.
So monitoring is part of testing. Track the distribution of key inputs and of the model’s scores against the training baseline, and alert when they move. Where the true outcome arrives later (a chargeback comes weeks after the transaction), compute precision and recall on that delayed ground truth and watch the trend. Sensible rollout is gradual: run the new model in shadow mode, scoring live traffic without acting on it, then compare against the current one before it takes decisions for a small share of users.
Retraining is a release too. A retrained model is a new version and deserves the same slice checks, comparison with the previous model and rollback plan as any other change, because “same code, new data” is a perfectly good way to break something.
Figures and names in the examples are illustrative.
Leave a Reply