Randomised experiments are the cleanest way to learn whether something causes something else, but many business questions arrive after the fact, or involve decisions nobody would randomise. Does Gold status make customers more loyal? Does using the mobile app increase spend? You cannot randomly hand out Gold status or force people to install an app. What you can sometimes find is a rule or a nudge in your existing data that behaved almost like a random assignment. Two techniques make use of these: regression discontinuity and instrumental variables.
Regression discontinuity: the power of a cut-off
Imagine a retailer whose loyalty scheme grants Gold status to anyone who spends ₹50,000 or more in a calendar year. Gold members get priority support and extra discounts. Comparing all Gold members with all non-Gold members is useless for measuring the effect of Gold status, because Gold members were bigger spenders to begin with.
But consider two narrow groups: customers who spent between ₹49,000 and ₹49,999, and those who spent between ₹50,000 and ₹50,999. These customers are nearly identical in shopping behaviour. Whether a particular person landed just above or just below the line depends on small accidents: one extra phone case, a delayed return. Yet one group got Gold and the other did not. If next year’s spend jumps sharply right at the ₹50,000 mark, that jump is a credible estimate of Gold status’s effect for customers near the threshold.
In practice, you plot the outcome (next-year spend) against the running variable (this-year spend), fit a line or smooth curve on each side of the cut-off using only data within a chosen bandwidth, and measure the gap between the two lines at the threshold. Real-world cut-offs are everywhere: credit score thresholds for loan approval, age limits for products, minimum order values for free delivery, exam score cut-offs, eligibility limits for government schemes.
When the cut-off stops being “as good as random”
The method relies on people not being able to place themselves precisely on one side of the line. Loyalty thresholds often fail this test. Customers who know they are at ₹48,500 in late December may deliberately buy something to cross ₹50,000. Those who make that effort are probably more engaged than those who do not, so the group just above the threshold is no longer comparable to the group just below.
The tell-tale sign is bunching: an unusual pile-up of customers just above the cut-off and a dip just below. A density test (the McCrary test is a well-known one) formalises this check. If there is clear manipulation, the simple comparison is biased. Cut-offs that people cannot see or control, such as an internal risk score, tend to work much better.
Two other caveats. The estimate applies to people near the threshold; it says little about the effect of Gold status on someone spending ₹3 lakh. And results can be sensitive to the bandwidth and the curve fitted, so report a few reasonable choices.
Instrumental variables: using a nudge to isolate an effect
Now the app question. App users spend more than web-only users, but people who install the app were already more engaged, so the raw gap is inflated. Suppose, though, that six months ago the CRM team ran a randomised campaign: half of web customers received an SMS encouraging them to install the app, half did not. Nobody was forced to install, but the SMS changed the odds.
Illustrative results:
- SMS group: 30% installed the app; average spend over the next quarter ₹2,240.
- No-SMS group: 10% installed the app; average spend ₹2,000.
Because the SMS was randomised, the ₹240 difference in average spend is a fair estimate of the effect of sending the SMS. If the SMS affects spend only by getting people to install the app, then the ₹240 came entirely from the extra 20 percentage points of installers. Dividing gives the effect of the app on those people: ₹240 ÷ 0.20 = ₹1,200 per additional installer. The SMS here is the instrument: something that shifts the treatment but has no other route to the outcome.
This simple ratio is called the Wald estimator. With covariates and continuous instruments, the same logic is implemented through two-stage least squares.
The assumptions, and why they bite
An instrument has to satisfy three conditions:
- Relevance. It must genuinely move the treatment. A nudge that raised installs by half a percentage point would produce an unstable, noisy estimate (the “weak instrument” problem). This one is testable.
- As-good-as-random assignment. Easy when the instrument was randomised, much harder when it is something observational like distance to a store or a policy change.
- Exclusion. It must affect the outcome only through the treatment. If the SMS also carried a ₹100 coupon, it would raise spend directly, and the ₹1,200 would be overstated. This cannot be tested statistically; it must be argued from knowledge of how the instrument works.
The estimate also has a specific meaning. It applies to “compliers”: people who installed because of the SMS and would not have otherwise. Enthusiasts who install regardless, and holdouts who never install, contribute nothing to it. This is called the local average treatment effect. It may be exactly the right number for deciding whether to run more install campaigns, since those campaigns would reach similar marginal users, and the wrong number for estimating the value of the app’s existing power users.
Spotting opportunities
Both methods depend on noticing structure already in your business. Look for hard thresholds in eligibility rules, scoring models and pricing slabs; for past randomised campaigns that nudged behaviour without forcing it; and for operational quirks such as which agent or warehouse handled a case, when assignment was effectively arbitrary. When you find one, you may be able to answer a causal question with data you already hold. When the assumptions look shaky, say so, and treat the result as suggestive rather than proof.
Written by the RP Strativa Team. The retailer, thresholds and figures are illustrative.
Leave a Reply