Two inside-sales teams at a B2B software company each handled 500 leads last quarter. Team B closed 288 of them, a conversion rate of 57.6%. Team A closed 130, just 26%. The sales head is ready to move Team A’s manager to a different role. Then an analyst splits the leads by source.

Lead sourceTeam ATeam B
Inbound (they asked for a demo)90 of 120 = 75.0%280 of 400 = 70.0%
Outbound (cold prospects)40 of 380 = 10.5%8 of 100 = 8.0%
All leads130 of 500 = 26.0%288 of 500 = 57.6%

Team A converts better on inbound leads and better on outbound leads. Team B converts better overall. Nothing is miscalculated. This reversal is Simpson’s paradox, and it is one of the most practical ideas in all of statistics because it turns up constantly in real business data.

How the reversal happens

An overall rate is a weighted average of the segment rates, and the weights are each group’s share of volume in each segment. Team B received 400 inbound leads out of 500, so 80% of its work was in the easy segment where everyone converts well. Team A received only 120 inbound leads; 76% of its work was cold outreach, where conversion is around one in ten for anyone.

Team B’s high overall rate is mostly a statement about what it was given, not how well it worked. Team A was better at both kinds of lead, but it spent most of its time in the hard segment, and its average was dragged towards that segment’s low rate.

Two conditions together produce the paradox. First, the segments have very different baseline rates (75% versus about 10% here). Second, the groups being compared have very different mixes across those segments. Remove either condition and the reversal cannot occur. If both teams had the same share of inbound leads, the better team in each segment would also be better overall.

Where it shows up in practice

  • Hospitals and doctors. A specialist hospital can show higher overall mortality than a general one while having better outcomes for every severity level, simply because it receives the most critical cases.
  • Branch and region comparisons. A bank branch in a business district may show worse loan default rates than a suburban branch overall, yet better rates within each loan type, because it writes far more unsecured business loans.
  • Month-on-month metrics. Average order value can fall even though it rose in every category, because the mix shifted towards a cheaper category. A festive sale pulling in many first-time, low-value buyers does exactly this.
  • A/B tests with uneven traffic. If a test’s traffic split changes partway through, and conversion also changes over time (weekdays versus weekends, salary week versus month-end), pooled results can disagree with every daily comparison.
  • Admissions and hiring. The best-known historical case is the 1973 graduate admissions data at the University of California, Berkeley, which appeared to show bias against women overall. Department by department the picture was different, largely because women applied in greater numbers to departments that admitted a smaller share of all applicants.

So which number is right?

Here is where people get stuck. The paradox does not mean “always use the segmented numbers”. The right answer depends on the causal story behind the data, and that is a question about the business, not about the arithmetic.

In the sales example, lead source was decided before the teams did anything. It is a pre-existing condition that affects both which team got the lead and how likely the lead was to close. Statisticians call such a variable a confounder. When you want to judge the team’s skill, you should compare within lead source, because you want to remove the effect of an assignment the team did not control. Team A is the stronger team.

Now change the story. Suppose a new onboarding flow is tested, and the segmenting variable is “did the user complete the tutorial”. If the new flow causes more people to complete the tutorial, and tutorial completion drives conversion, then splitting by tutorial completion hides exactly the effect you are trying to measure. The variable sits on the causal path, it is a consequence of the treatment, and here the aggregate is the right comparison.

The rule of thumb: segment by variables that were fixed before the thing you are evaluating happened and that influence the outcome. Do not segment by variables that the thing you are evaluating itself changes.

Habits that catch it early

Whenever you compare rates between two groups, check the mix before the rates. A quick table of “share of volume by segment” for each group takes a minute and reveals whether the comparison is like-for-like. If the mixes differ a lot, compute the comparison within segments, or produce a mix-adjusted figure: apply each group’s segment rates to a common mix. Using an equal 50-50 split of inbound and outbound for both teams, Team A comes out at about 42.8% and Team B at 39%, which puts them in the right order.

When you present the result, show both the total and the segments, and explain the mix difference in one sentence. Executives do not need the name “Simpson’s paradox”. They need to hear “Team B looks better overall because it was handed most of the warm leads; on comparable leads Team A wins”. That sentence prevents a bad people decision, which is the whole point of the analysis.

Written by the RP Strativa Team. The company, teams and figures in the sales example are illustrative.

Leave a Reply

Trending