The belief: if a test shows a significant result at the 5 percent level, there is only a 5 percent chance it is a fluke. It is a tidy belief, and it holds for exactly one situation, a single comparison decided in advance. Change that, and the 5 percent quietly stops meaning what you think.

The arithmetic of twenty looks

Suppose a product team ships a new home screen, and the experiment dashboard reports twenty metrics: clicks, add-to-cart, orders, revenue, time on page, search use, and so on. Assume, for the sake of argument, that the new screen changes nothing at all. Every metric has a 5 percent chance of crossing the significance line by luck, since that is what the threshold means when there is no real effect.

The chance that at least one of twenty independent metrics comes out significant is 1 minus 0.95 to the power of 20, which works out to about 64 percent. More likely than not, a screen that does nothing will show something. Somebody will spot the one green number, write “add-to-cart up 3.1 percent (p = 0.04)” on a slide, and the launch is approved on noise.

Real metrics are correlated, so the true figure is lower than 64 percent, but the direction holds, and the same issue arises when you test five variants against a control, or slice a result by city, device and customer segment until something lights up. Each slice is another ticket in the lottery.

Two ways to control it

The conservative route controls the family-wise error rate, the probability of even one false positive across all tests. The simplest method is Bonferroni: divide the threshold by the number of tests. With twenty metrics, each must pass at 0.05 / 20 = 0.0025. Run the numbers again and the chance of at least one false positive when nothing is going on is about 4.9 percent, back to where you wanted it.

The price is power. A stricter threshold per test means real, modest effects will be missed more often. Bonferroni is also known to be conservative when tests are correlated. Holm’s step-down method is uniformly a little more powerful and is just as easy to apply, so it is usually the better choice when you want strict control.

The other route controls the false discovery rate: among the results you call significant, what share are expected to be false? The Benjamini-Hochberg procedure does this. You rank the p-values from smallest to largest and compare each to a rising threshold, rank divided by number of tests, times your chosen rate. It tolerates a few false alarms in exchange for finding more real effects, which suits exploratory work where findings will be followed up. For a go or no-go decision that costs real money, the stricter control is safer.

The better fix happens before the test

Statistical corrections rescue a badly designed analysis. A well-designed one avoids needing them.

  • Choose one primary metric before launch, the one the decision depends on. It gets the full 5 percent.
  • Declare a short list of secondary metrics and a separate set of guardrails (things that must not get worse, such as page load time or refund rate). Treat these as supporting evidence, and apply a correction if any of them can drive a decision.
  • Write the plan down, including which segments you will look at. Segments chosen after seeing the data are hypotheses, not findings.
  • If you test several variants against one control, correct for the number of comparisons, or run a combined test first and only compare pairs if it is significant.

What to do with a surprise

An unexpected green number is not worthless. It is a lead. Say it plainly: “Among the nineteen unplanned metrics, one moved; we did not plan for it, so we will treat it as a hypothesis and test it directly.” Then run a follow-up experiment where that metric is the primary one. If the effect is real, it will show up again. If it was chance, it will fade, and you will have saved yourself a launch based on it.

Finally, remember that the opposite worry is also real. Correcting too aggressively across dozens of exploratory metrics can bury a genuine effect. The sensible middle is small: a clear primary metric, a short honest list of others, and an agreed rule for the rest.

Leave a Reply

Trending