A smoke detector can fail in two ways. It can shriek when you are only making toast, or it can stay silent while the kitchen fills with smoke. You can make it less jumpy, but then it is more likely to miss a real fire. You can make it more sensitive, but then it goes off every morning. Statistical tests face exactly this trade-off, and the two failures have names.

The two errors

A test starts from a null hypothesis, the “nothing is happening” position: the new pricing page converts no better than the old one, the new supplier’s defect rate is the same, the campaign did not move sales. The test then either rejects that null or fails to reject it. Reality also has two states. That gives four combinations.

No real effectReal effect exists
Test says “effect”Type I error (false positive)Correct detection
Test says “no evidence”CorrectType II error (false negative)

A Type I error is the toast alarm: you conclude the new pricing page works when it does not. Its probability, when the null is true, is the significance level alpha, which you choose. At the usual 0.05, you accept that one in twenty tests of a useless change will look like a winner.

A Type II error is the silent detector: a real improvement exists, but your test fails to flag it. Its probability is called beta. The complement, one minus beta, is the test’s power: the chance of detecting an effect of a given size if it is really there. A common target is 80% power, meaning a one-in-five chance of missing a real effect of the size you planned for.

Which mistake costs more?

The conventional 5% and 80% are defaults, not laws. The right balance depends on what each error costs in your situation.

  • Fraud screening. A Type II error lets a fraudulent transaction through; a Type I error blocks a genuine customer and annoys them. Banks usually tolerate many more false alarms because a missed fraud costs more, then use a second check to clear the false alarms.
  • Launching a costly change. If rolling out a new warehouse process means retraining 2,000 staff, a Type I error (adopting something that does not help) is expensive. A stricter alpha, say 0.01, makes sense.
  • Cheap, reversible tweaks. For a button colour that can be reverted in five minutes, a false positive costs little. Missing a real gain is the bigger loss, so you might accept a looser alpha and push for higher power.
  • Medical and safety testing. A screening test for a serious disease is tuned to avoid missed cases, accepting that some healthy people will be called back for further tests.

For a fixed amount of data, lowering one error raises the other, just like the smoke detector. The only way to reduce both at once is to collect more data or reduce noise.

Power is decided before the test, not after

Power depends on four things: the significance level, the sample size, the variability of the data, and the size of the effect you want to detect. Fix three and the fourth follows. That is how sample sizes are planned.

Take a pricing page that converts at 4%. The team wants to detect an improvement to 4.8%, a 20% relative lift, using a two-sided test at alpha 0.05 with 80% power. The standard formula for comparing two proportions gives roughly 10,300 visitors per variant, so about 20,600 in total. If the page gets 1,500 visitors a day, that is around two weeks.

Now suppose the team says a 10% lift, to 4.4%, would also be worth knowing about. Halving the detectable effect roughly quadruples the required sample, because sample size scales with one over the square of the effect. The figure jumps to about 39,500 per variant, nearly two months of traffic. Asking for 90% power instead of 80% at the original 20% lift raises it from about 10,300 to about 13,800 per variant.

These numbers explain why many A/B tests in smaller Indian startups and B2B products are underpowered. With a few hundred conversions a month, only large effects are detectable in a reasonable time, and subtle wins are simply invisible to the test.

What underpowered tests do to your decisions

The obvious cost of low power is missed wins: real improvements get labelled “no significant difference” and are abandoned. Teams then conclude that “nothing we try works”, when in fact their tests could not have seen what worked.

The less obvious cost is inflated winners. When an underpowered test does cross the significance line, it is often because random noise pushed the observed effect upward. The results that get through the filter tend to be overestimates, a pattern sometimes called the winner’s curse. A test designed to detect a 20% lift that comes back showing 35% should make you more cautious, not less. Expect the effect in production to be smaller.

Low power also makes the share of false positives among your “wins” higher. If only a small fraction of ideas truly work, and your tests catch only a fraction of those, then the 5% false alarms from the many ideas that do not work can make up a large share of everything declared significant.

Practical habits

  1. Agree the minimum effect worth detecting with the business before the test. “What lift would change our decision?” is a business question, not a statistical one.
  2. Calculate the sample size from that effect, and commit to it. Stopping early because results look good raises the Type I error rate.
  3. If the required sample is unrealistic, change the plan: test a bolder change, use a metric closer to the change with less noise, or accept a lower power and say so.
  4. When a test finds nothing, report what it could have detected. “No significant difference; this test had 80% power to detect a lift of 20% or more” is honest. “The change has no effect” is not.

Most of the value in thinking about Type I and Type II errors comes before any data is collected. Once you know which mistake hurts more and how big an effect matters, the test design almost writes itself.

Written by the RP Strativa Team. Figures in the examples are illustrative.

Leave a Reply

Trending