A fictional product manager in Bengaluru, call her Meera, is asked to find out what makes customers happy with a budgeting app. She pulls ratings from the app’s in-product survey, which appears after a user has been active for 60 days. The average is 4.4 out of 5. Users who set up three or more budgets rate it even higher. Her recommendation writes itself: push every new user to create three budgets in the first week.
What is wrong with this? Think about who was ever asked. Anyone who uninstalled in week two never reached day 60 and never saw the survey. The people in her data are the ones who stayed, and people who stay tend to like the product. Of course their ratings are high. Her data cannot say anything about the people who left, and those are exactly the people whose behaviour she needs to understand.
The planes that came home
The classic telling involves the statistician Abraham Wald during the Second World War. Military analysts studied bombers returning from missions and mapped where the bullet holes were. The obvious idea was to add armour where the holes clustered. Wald’s insight, as the story is usually told, was that the data came only from planes that survived. Holes in the returning planes marked places where a hit was survivable. The missing planes were the ones hit elsewhere, and those were the places needing armour. Details of the retelling vary, but the reasoning is sound and worth keeping.
How it inflates numbers, with arithmetic
Imagine 100 mutual funds launched in a category ten years ago. Over the decade, 20 of them performed badly and were merged into other schemes or closed. Suppose the 80 survivors averaged 12 percent a year and the 20 that disappeared averaged 2 percent a year. The true average across all 100 is (80 x 12 + 20 x 2) / 100 = 10 percent. A database that lists only today’s live funds reports 12. The two-point gap is not noise. It is built into how the list was assembled, and it grows when the casualties are more numerous.
The same shape appears across business data:
- Analysing customers who are active today to learn what retains customers.
- Studying successful start-ups for common habits, without looking at the many that followed the same habits and failed.
- Computing average tenure of employees currently on payroll, which excludes everyone who has already left and so overstates how long people stay.
- Back-testing a trading rule on a stock index’s current constituents, when the index quietly dropped the losers over the years.
- Reading reviews on a product page, where unhappy buyers may have returned the item and never reviewed it.
Back to Meera
What would a better analysis look like? She would start from everyone who signed up in a given month, not everyone who is active now. This is a cohort, and it includes the ones who left. For each person she asks: did they create three budgets in week one, and were they still using the app at day 60? Now she is comparing survivors with non-survivors on the same footing.
Say that out of 1,000 sign-ups, 300 created three budgets early and 700 did not. If 150 of the 300 remain at day 60 and 140 of the 700 remain, retention is 50 percent against 20 percent. That is a real gap, though still not proof that budgets cause retention; motivated users may simply do both. But at least the data now contains the people who left, and the argument can move forward honestly.
Questions to ask any dataset
The habit to build is a single question: how did these rows get into this table? Whoever or whatever filtered the population before it reached you, a survey trigger, a “status = active” flag, a CRM that deletes closed accounts, a vendor that only reports live funds, has shaped your answer before you wrote a line of analysis.
A few practical checks. See whether deleted, closed, churned or inactive records are kept somewhere, perhaps in an archive table or a soft-delete column. Compare the count of records at the start of a period with the count you can account for at the end. Look for the unusual silence: a segment that you would expect to see, absent. And when the only available data is survivors, say so in the write-up and soften the claim. “Among users who stayed 60 days, three budgets correlated with higher ratings” is true. “Three budgets make users happy” is not supported.
The bias is hard to see precisely because absent data does not announce itself. A table never has a row saying “this customer left before you could ask”.
Figures and names in the examples are illustrative.
Leave a Reply