Pull up a chair. Think about the one classmate you had in school who memorised the whole guide book, including the sample answers. Put him in a test that reused the guide’s questions and he topped the class. Change the wording slightly and he was lost. He had not learned the subject. He had learned the paper.
A predictive model can do exactly this, and the name for it is overfitting. Once you see it as “memorising the exam”, most of the standard machinery around model building starts to make sense.
What the model is really trying to learn
Say you have 5,000 past orders from a food-delivery business and you want to predict which new orders will be returned or refunded. Each row has the order value, the hour, the restaurant, the delivery distance and so on. Among those 5,000 rows there is real signal (long delays do cause refunds) and there is noise (a customer who was in a bad mood, a rider who took a wrong turn once). Your model cannot tell the two apart. It only sees rows.
A simple model, say a short list of rules, has little room to wiggle, so it can only capture the broad pattern. A very flexible model, say a decision tree grown until every leaf holds one order, has enough room to carve out a special rule for every odd case. “Orders above ₹640 placed at 9:17 pm from this restaurant with this distance were refunded” is a perfectly valid rule for the training data. It is also useless next week.
So there are two ways to be wrong. Underfitting means the model is too simple to catch even the real pattern. Overfitting means it has also absorbed the noise. The first shows up as poor results everywhere. The second is sneakier, because the results look excellent where you are looking.
Why you can’t mark your own homework
If you measure accuracy on the same rows the model learned from, you will always be flattered. The more flexible the model, the more flattered you are. A tree grown to full depth can score near-perfectly on its own training data and still be mediocre on new data. That is why the standard habit is to keep some rows aside before you start.
Split the 5,000 orders into 4,000 for training and 1,000 for testing. The model never sees the 1,000 while it is being built. After training you score both sets and compare. A picture you might see, with illustrative numbers:
| Model | Accuracy on training rows | Accuracy on held-out rows |
|---|---|---|
| Three simple rules | 78% | 77% |
| Moderate tree | 86% | 83% |
| Tree grown to the bottom | 99% | 74% |
Look at the gap, not just the level. The first model is honest but modest. The last one looks like a champion on the training sheet and is actually the worst of the three. The middle one is what you would ship. The size of the gap between training and held-out performance is your best quick indicator of overfitting.
The test set is single-use
Here is a subtle point that trips up people who already know the basic split. Suppose you try thirty different model settings and pick whichever scores best on your 1,000 held-out rows. You have now used those rows to make a decision, so they are no longer a clean exam. Your chosen model is partly chosen because it happened to suit that particular 1,000.
The usual remedy is three-way: a training set to fit models, a validation set to compare and tune them, and a final test set you touch once, at the end, to report how good the chosen model is likely to be. When data is scarce, people use cross-validation instead. You cut the training data into, say, five parts, train on four and check on the fifth, rotate five times, and average. It costs more computing but wastes fewer rows.
Splitting randomly is not always fair
A random split assumes tomorrow looks like a shuffled version of yesterday. For anything that moves with time, it doesn’t. If you are predicting next month’s refunds, a random split lets the model train on orders from after some of the orders it is tested on. The honest version is to train on the older months and test on the newest ones, which mimics how the model will actually be used. If your test accuracy falls noticeably under a time-based split compared with a random one, something in your features is date-sensitive, such as a festival season, a pricing change or a new city launch.
Another trap is grouped data. If one customer appears in forty rows, and some of those rows land in training and some in testing, the model can recognise the customer instead of learning the pattern. Split by customer, not by row.
What you can do about it
- Get more data. Noise averages out when there is more of it, so this works better than any trick.
- Simplify. Fewer features, shallower trees, fewer interaction terms.
- Regularise. Techniques such as ridge and lasso regression add a penalty for large coefficients, which nudges the model toward plainer explanations.
- Stop early or prune. Many algorithms let you cap tree depth or stop training when validation results stop improving.
You will also hear “bias” and “variance” in this context. Bias is the error from being too rigid (underfitting). Variance is the error from being too sensitive to the particular sample you trained on (overfitting). Reducing one tends to raise the other, so the job is finding a sensible middle, not eliminating either.
One honest caveat. A good held-out score tells you the model works on data that resembles your history. It says nothing about a world that has changed. A refund model built before a new payment partner went live may fail not because it overfitted but because the ground moved. That is why teams keep monitoring a model after launch rather than trusting the number it had on test day.
Figures and names in the examples are illustrative.
Leave a Reply