A/B test significance

The same A/B test read two ways: a p-value with its confidence interval, and the posterior probability that the variant is actually better.

Control

4.00% conversion rate

Variant

4.28% conversion rate

Used for both intervals.

Applies to the p-value.

%

Where the change pays for itself.

Control rate
4.00%
1,600 of 40,000
Variant rate
4.28%
1,712 of 40,000
Observed lift
+7.0%
+0.28 pp in absolute terms

Frequentist

significant at 5%
p-value
0.0468
z = 1.988
95% CI on relative lift
+0.1% to +14.4%
Katz log method
95% CI on absolute difference
+0.00 pp to +0.56 pp
percentage points

If the two arms really converted at the same rate, a gap at least this large would turn up 4.7% of the time. That is not the probability that the variant is better, and the interval is not a 95% probability that the true lift sits inside it — repeat this experiment many times and 95% of the intervals built this way would cover the truth.

Bayesian

P(variant is better)
97.6%
2.4% chance it is worse
95% credible interval
+0.1% to +14.3%
on the relative lift
P(lift > 5%)
70.7%
clears the bar you set above
Expected loss from shipping
0.029%
of conversions, averaged over everything that could be true
If it does turn out worse
−1.21%
average damage across the losing draws

Given this data and a uniform prior that rules nothing out, there is a 97.6% probability the variant converts better than the control, and a 70.7% probability it clears the 5% lift you said it needs to be worth shipping. Those are statements about the lift, which is what the frequentist panel cannot give you.

What we now believe each conversion rate isControlVariant
3.60% 3.80% 4.00% 4.20% 4.40% 4.60% conversion rate Control Variant
How big the lift might be
0% +5% +10% +15% relative lift over control no change
variant is worse: 2.4% of the distribution 95% credible interval: +0.1% to +14.3%
Observed and posterior conversion rates for each arm
ArmObservedPosterior median95% interval
Control4.000%4.002%3.814% – 4.199%
Variant4.280%4.281%4.085% – 4.480%

The frequentist side is a pooled two proportion z-test, with the interval on the relative lift built by the Katz log method. The Bayesian side puts a Beta prior on each arm, adds the data to get Beta(1,601, 38,401) and Beta(1,713, 38,289), then takes 30,000 Monte Carlo draws to get the lift. The draws are seeded, so the same inputs always give the same answer.

Both panels assume one look at the end of the test. Watching either number until it crosses a line will ship losers more often than the number suggests.

Planning a test rather than reading one? Sample size calculator.