What Is Statistical Significance in A/B Testing?
What significance actually means
Statistical significance is the answer to one question: if the two variants were actually identical, how likely is it that you'd see a difference this big by chance alone? That probability is the p-value. When it's small enough, you conclude the difference is unlikely to be luck and call the result significant.
The threshold you compare against is the significance level (alpha), most commonly 0.05. Its mirror image is the confidence level: a 0.05 alpha is the same as 95% confidence. A 95% significant result means there's roughly a 1-in-20 chance you're being fooled by randomness.
The formula (two-proportion z-test)
For comparing two conversion rates, the standard tool is a two-proportion z-test. It produces a z-score, a measure of how many standard errors apart the two rates are. The bigger the z-score, the less likely the gap is chance.
where p̂ = (x₁ + x₂) ÷ (n₁ + n₂) is the pooled conversion rate
Here p₁ and p₂ are the two conversion rates, n₁ and n₂ the visitors in each variant, and x₁/x₂ the conversions. A z-score of about 1.96 corresponds to 95% confidence (two-tailed); 2.58 corresponds to 99%.
A worked example
Say variant A converts 100 of 2,000 visitors (5.0%) and variant B converts 130 of 2,000 (6.5%).
- Pooled rate p̂ = (100 + 130) ÷ 4,000 = 0.0575
- Standard error = √[0.0575 × 0.9425 × (1/2000 + 1/2000)] ≈ 0.00736
- z = (0.065 − 0.050) ÷ 0.00736 ≈ 2.04
A z of 2.04 is just past 1.96, so this result clears 95% confidence (p ≈ 0.041). You can reasonably conclude variant B is the genuine winner, though the margin is thin, so more data would make the call more solid.
How to interpret the result
Significance answers "is it real?", not "is it big enough to matter?" Always read it alongside two other things:
- Effect size / lift. A statistically significant 0.1% lift may not be worth shipping. Significant and meaningful is the bar.
- Confidence interval. A range like "+0.5% to +2.5%" is more honest than a single number. If the interval crosses zero, you don't have a winner.
Common mistakes
| Mistake | Why it bites | Fix |
|---|---|---|
| Peeking and stopping early | Inflates false positives well above 5% | Set a sample size up front; run to it |
| Tiny samples | Underpowered tests miss real effects | Check required sample size first |
| Confusing significance with impact | Ships trivial "wins" | Require a minimum lift too |
| Running many variants at once | More comparisons = more false positives | Correct alpha or limit variants |
Frequently asked questions
What significance level should I use for A/B tests?
Most teams use a 95% confidence level (a 0.05 threshold), a 5% accepted false-positive rate. Use 90% for low-risk tests where speed matters, and 99% for high-stakes changes like pricing or checkout.
Is a 95% significant result guaranteed to be correct?
No. 95% significance still leaves about a 1-in-20 chance of a false positive, and it says nothing about whether the effect is large enough to matter, only that it's unlikely to be pure chance.
Can I stop an A/B test as soon as it hits significance?
No. Stopping the moment a test crosses 95%, a habit called "peeking", inflates your false-positive rate dramatically. Decide a sample size in advance and run the test to that size before reading the result.
Last updated: 14 June 2026