An online retailer sells a paid membership at ₹999 a year, offering free delivery and early sale access. A year later, the numbers look spectacular: members spent an average of ₹6,200 in the following twelve months, non-members ₹2,900. The membership team wants to report that the programme adds ₹3,300 of spend per customer.

The problem is obvious once stated. Customers who chose to pay for membership were already the heavy buyers. They signed up because they expected to shop a lot. Comparing them with everyone else measures the difference between heavy and light shoppers, not the effect of membership. This is selection bias, and it affects nearly every analysis where people choose their own treatment: users who adopt a feature, employees who attend training, customers who respond to an offer.

The idea: compare like with like

The fix in principle is to compare each member with a non-member who looked the same before joining: same past spend, same order frequency, same tenure, same city, same category preferences. If two customers were identical on everything that drives both joining and spending, and one joined while the other did not, the difference in their later spend is a fair estimate of membership’s effect.

Matching on many variables at once quickly becomes impossible, because exact twins across ten characteristics are rare. Paul Rosenbaum and Donald Rubin showed in 1983 that you can instead match on a single number: the propensity score, the probability that a customer would join given their observed characteristics. Two customers with the same propensity score are, on average, balanced on the characteristics used to compute it, even if they differ on individual ones.

The steps

  1. Choose covariates measured before treatment. Prior-year spend, orders, returns, tenure, city tier, device, categories bought. Never include anything measured after joining, such as spend in the first month of membership, because that is partly an outcome of the treatment.
  2. Model the probability of treatment. Usually a logistic regression (or a more flexible model) predicting membership from those covariates. The predicted probability is each customer’s propensity score.
  3. Match. For each member, find one or more non-members with a very similar score, often within a set tolerance called a caliper. Members with no close match are dropped.
  4. Check balance. Compare the matched groups on every covariate. Standardised mean differences close to zero (a commonly used threshold is below 0.1) suggest the matching worked. If the groups still differ, revise the model.
  5. Compare outcomes. Only now look at later spend in the matched groups.

In our illustrative case, suppose the matched non-members, who had similar prior spend and habits, went on to spend ₹5,400. The estimated effect of membership is then ₹6,200 − ₹5,400 = ₹800, about a quarter of the naive figure. That is still a real effect, but it leads to very different decisions about how hard to push membership.

Note what the estimate refers to. Because we matched members to similar non-members, it is the average effect on the kind of customers who joined (often called the effect on the treated). It does not tell you what membership would do for light shoppers who never considered it.

The limitations that matter

It only adjusts for what you measured. This is the big one. If customers joined because they had just moved into a new flat and were about to buy lots of furniture, and you have no data on that, matching cannot fix it. Matched non-members would look similar on paper but lack the intention that drove both joining and spending. No statistical technique can balance an unobserved variable. Randomised experiments are valuable precisely because randomisation balances unobserved factors too.

Overlap can be thin. If almost every customer spending above ₹50,000 a year joined, there are no comparable non-members for them. Those members get dropped, and your estimate silently stops describing the highest-value segment. Always report how many treated units were matched and plot the score distributions of both groups to see where they overlap.

Results depend on choices. Caliper width, one-to-one versus one-to-many matching, with or without replacement, and the covariate list all change the answer. Report the main estimate alongside a few reasonable alternatives. If the effect swings from ₹200 to ₹1,500 with modest changes, the honest conclusion is uncertainty.

Matching on the score itself has critics. Gary King and Richard Nielsen argued in a 2019 paper that matching on propensity scores can increase imbalance in some circumstances, and recommended other matching methods. Many practitioners now use alternatives such as inverse probability weighting, coarsened exact matching or doubly robust estimators, which combine a model of treatment with a model of the outcome. The underlying idea is the same: adjust for observed differences between groups before comparing.

How to present it

When sharing a matched comparison with leadership, show three things: the naive difference, the matched difference, and the balance table proving the matched groups look alike on the observed variables. Then list, in plain words, what was not observed and could still bias the result. A finding like “₹800 per member, assuming no unmeasured differences in purchase intent” is less exciting than ₹3,300, but it is the number that survives scrutiny, and it tells the business what it actually needs to know before scaling the programme.

Written by the RP Strativa Team. The retailer and figures in the examples are illustrative.

Leave a Reply

Trending