Field notes. Before I look at any A/B result, I look at the headcount. Not the conversion rate, not the p-value. How many users landed in each arm? If the answer is not what the design said, I stop. This is called sample ratio mismatch, SRM for short, and in my opinion it is the most under-checked failure in experimentation.
What it is
You configure a 50/50 split. You get 100,000 users. You expect about 50,000 in control and 50,000 in treatment. Randomness will not give you exactly that, so how far off is too far off?
Say you see 50,900 in control and 49,100 in treatment. That is a 1.8 percentage point gap in absolute counts and it looks tiny. It is not. A chi-square goodness-of-fit test against the expected 50,000 each gives a statistic of 32.4. With one degree of freedom that corresponds to a p-value of roughly one in eighty million. A fair coin does not do this. Something in the system is sending users to one side more than the other.
Now compare 50,500 and 49,500. The statistic is 10.0 and the p-value is about 0.0016. Still suspicious. Many teams use a strict threshold such as 0.001 for SRM alerts so that they do not cry wolf on every experiment, which is a sensible convention but only a convention. Pick yours and write it down before the test.
Why a skewed split ruins the result
The mismatch itself is not the harm. It is a symptom. It means users were dropped or added differently in the two arms, and if the reason correlates with behaviour, the two groups are no longer comparable. Randomisation was the thing that made the difference between arms attributable to your change. Once that is broken you are comparing apples to something that looks like apples.
Picture a checkout redesign whose new page is a bit heavier. On slow 4G connections in smaller towns, some users close the tab before the page loads and before the assignment event is logged. Those users vanish from treatment only. The remaining treatment group is faster-connection, higher-intent users, and conversion looks better than it really is. The SRM check would have flagged the missing users. The conversion lift would have celebrated them.
Where mismatches usually come from
- Logging asymmetry. The exposure event fires after a page or component loads, and the variants load at different speeds or fail at different rates.
- Bot and internal-traffic filters applied after assignment that catch one variant more than the other, for instance because the new variant triggers a script that bot detection dislikes.
- Redirect-based tests, where the treatment sends users to a new URL and some never arrive.
- Bucketing that depends on something that changes mid-test, such as a cookie that clears, or users who switch from logged-out to logged-in and get re-randomised.
- A ramp-up that was changed halfway (10 percent, then 50 percent) with the analysis pooling all days together.
- Data pipeline joins that silently drop users with missing attributes in one arm.
What to do when it fires
Do not adjust the numbers and carry on. Do not reweight to fake a 50/50. The honest response is investigation. Slice the mismatch by device, browser, country, app version and day. Mismatches rarely spread evenly; they concentrate. If 90 percent of the gap sits in one older Android browser, you have found your leak. Fix the cause, then rerun the test, because the data you have is not trustworthy.
One more opinion. Run the check automatically, in the experiment platform, for every test and every day. A check that depends on an analyst remembering is a check that will be skipped on the Friday before a launch decision. And apply it to the unit of analysis you actually use. If you randomise by user but count sessions, check users.
A caution about the test itself
The chi-square test is only valid when the expected counts are reasonably large and the units are independent. If you randomise by household or store but count individual users, the independence assumption fails and the test will fire too often. Also, with very large samples, a tiny harmless imbalance will become statistically significant. If you have 20 million users and a split of 50.02 to 49.98, check whether the cause is understood before throwing the experiment away. Most of the time, though, a significant SRM is exactly what it looks like: a broken experiment.
Figures and names in the examples are illustrative.
Leave a Reply