Take one test and give it to two analysts. A checkout page test: version A converted 120 of 2,400 visitors (5.00%), version B converted 138 of 2,400 (5.75%). The product manager wants to know if B is better.
The first analyst runs a standard two-proportion test and reports a p-value of about 0.25. “Not significant. Keep A, or run longer.” The second runs a Bayesian analysis and reports “there is roughly an 87% probability that B is better than A.” The product manager, reasonably, asks which one is lying. Neither. They are answering different questions, and I will argue at the end which one I would want on my desk.
The frequentist reading
Frequentist testing starts by assuming the two versions are identical, then asks: if that were true, how often would random noise alone produce a gap at least this large? For these numbers the standardised gap (z) is about 1.15, and noise alone would produce something as big or bigger roughly 25% of the time. That is the p-value of 0.25. By the usual cut-off of 0.05, you cannot reject “no difference.”
Notice what it does not say. It does not say there is a 25% chance that B is no better. It is a statement about the data under an assumption, not about the hypothesis. The method’s strength is that it offers a clear rule with controlled error rates: if you decide in advance on a sample size and a threshold, then over many experiments you will wrongly declare a winner about 5% of the time when nothing is going on. A 95% confidence interval for the lift runs from about −0.5 to +2.0 percentage points, which tells the same story: the plausible range includes zero.
The Bayesian reading
The Bayesian analyst treats each version’s true conversion rate as an unknown with a probability distribution. Begin with a starting belief (a prior), update it with the data, and you get a posterior distribution for each. With a flat prior, both posteriors are tight humps around 5.0% and 5.75%, overlapping heavily. Simulating many draws from each and counting how often B’s draw is higher gives about 87.5% (I ran a million draws to check). The average lift is about 0.75 points, and a 95% credible interval is again roughly −0.5 to +2.0.
Here the sentence “87% chance B is better” is a legitimate thing to say, because the framework treats the unknown rate as something you can hold beliefs about. That is the number most stakeholders think a p-value gives them. It is a more natural answer, and that natural quality is the main argument for it.
Where they actually differ
| Point | Frequentist | Bayesian |
|---|---|---|
| What is random? | The data, over repeated experiments | The unknown rate, given the data you saw |
| Headline output | p-value, confidence interval | Probability B beats A, credible interval |
| Prior knowledge | Not formally used | Enters through the prior |
| Stopping early | Peeking inflates false positives unless corrected | More forgiving in principle, not immune in practice |
| Decision rule | Threshold on p, fixed sample size | Threshold on probability or expected loss |
With plenty of data and a weak prior the two usually agree on the decision, and they agree here: the evidence is inconclusive. What changes is the language. “Not significant” invites a lazy reading of “no effect”. “87% likely better, with a plausible range of −0.5 to +2.0 points” invites a conversation about cost and risk.
The claims that need pushing back on
Bayesian testing is often sold as a cure for peeking, and that is overstated. If you check daily and stop the moment the probability crosses 95%, you still push your error rate up. The mathematics is more gentle about it, but a stopping rule chosen after seeing the data is still a risk. Decide in advance what level of evidence ends the test, whichever framework you use.
The prior is the other soft spot. A flat prior is a choice, not neutrality. If you start with “any rate from 0% to 100% is equally likely” you are being unrealistically open-minded about a checkout that has always converted around 5%. A sensible informative prior from past tests speeds things up and shrinks wild early swings, but now two analysts can differ on the prior. With small samples the answer then depends on it. State the prior in the report.
On the frequentist side, the weakness is mostly human. Teams read p = 0.049 as victory and p = 0.051 as failure, though the evidence barely differs. The method is fine; the cult of the threshold is not.
Taking a side
For a business that runs many routine tests, I would lean Bayesian in how results are communicated, and frequentist in how the test is designed. Fix the sample size and stopping rule up front, which is frequentist discipline. Then report an interval and a probability of improvement, together with the cost of being wrong, which is Bayesian communication.
In the checkout example, the decision hinges on money more than on a probability. If B costs nothing to ship and the downside is small, an 87% chance of being better is plenty to switch. If B needs a vendor contract of ₹30 lakh a year, you want much more evidence than either analyst produced. A p-value alone cannot tell you that, and a posterior probability alone cannot either. Expected gain and expected loss can, and they come easily from a Bayesian analysis.
Whichever camp you join, the checks that matter are the boring ones: a correct split of traffic, a pre-agreed metric, and enough visitors. Statistics cannot rescue a broken experiment.
Figures and names in the examples are illustrative.
Leave a Reply