A widely repeated rule says that if less than 5% of values are missing, you can just drop those rows, and if more, fill the gaps with the column average. It is neat, easy to remember and wrong often enough to be dangerous. The percentage missing is the least interesting fact about a gap. The interesting fact is why it is there.
Three reasons a value goes missing
Statisticians, following Donald Rubin’s classification, sort missingness into three kinds.
Missing completely at random (MCAR) means the chance of a value being missing has nothing to do with anything, observed or not. A sensor loses power for an hour; a data-entry operator skips a random field. The remaining data is a fair, smaller sample of the whole. This is the harmless case, and also the rarest.
Missing at random (MAR), an unfortunate name, means the chance of being missing depends on other columns you do have, but not on the missing value itself once those are accounted for. Younger customers skip the “annual income” field more often than older ones, and you know their ages. The gaps are patterned, but the pattern is visible in your data and can be modelled.
Missing not at random (MNAR) means the chance of being missing depends on the value that is missing. High earners leave income blank because it is high. Patients with the worst symptoms miss follow-up visits. Customers who were unhappy with a delivery do not fill in the satisfaction survey. The cause sits in the very number you cannot see, and no amount of clever filling recovers it from the data alone.
A ten-customer demonstration
Ten customers have annual incomes, in lakh rupees: 4, 5, 6, 7, 8, 9, 12, 15, 20 and 30. The true average is 11.6. Now suppose the three highest earners leave the field blank, which is the MNAR pattern. You are left with seven values, and their average is about 7.3. Dropping blank rows gives you a figure 37% too low, and the spread looks much tighter than it truly is (standard deviation about 2.7 against a true 8.1).
Now fill the three blanks with 7.3, the mean of what you observed. The average stays at 7.3, still wrong, and you have manufactured three customers who all earn exactly the typical amount. Mean-filling also shrinks variance and weakens relationships with other variables, so correlations and model coefficients get quietly diluted. Neither common technique helps, because the problem was the mechanism, not the method.
What to do in each case
For MCAR with a small share missing, dropping rows (complete-case analysis) loses precision but not accuracy. It is often fine.
For MAR, use the other columns. Regression-based imputation, nearest-neighbour imputation, or multiple imputation (creating several plausible completed datasets, analysing each and combining the results so that the uncertainty is honestly carried through) all use the observed relationships. In the income example, if the missingness depended on age, filling within age bands would be far better than a global average.
For MNAR, you need outside information or an explicit assumption. Options include a follow-up survey of non-responders, a sensitivity analysis showing how conclusions change if the missing values are, say, 30% higher than the observed ones, or collecting the data differently next time. Honestly flagging the limitation is part of the answer.
Practical habits
- Add an indicator column such as “income_missing”. Often the fact of missingness predicts something useful.
- Compare rows with and without the value on other columns. If the two groups look different, it is not MCAR.
- Distinguish “unknown” from “not applicable” and from “zero”. A blank loan amount for a customer with no loan is not the same as an unreported one.
- Ask the people who run the source system. Many a “random” gap turns out to be a form change on a known date.
You cannot prove MAR against MNAR from the data alone, since the difference lies in values you do not have. So the classification is partly a judgement about how the data was produced, and the best evidence for it usually comes from understanding the business process, not from a test statistic.
Figures and names in the examples are illustrative.
Leave a Reply