What Is Statistical Confidence (95% Confidence)?
What statistical confidence means
Statistical confidence is a statement about your method, not about a single result. A 95% confidence level means that if you ran your test process over and over on data where nothing had actually changed, you would wrongly declare a "winner" only about 5% of the time. The confidence level is the long-run rate at which the procedure avoids being fooled by random noise.
In A/B testing it's the number you set before the test runs: the bar a result must clear before you trust it. The higher you set it, the rarer false positives become, but the more data you need to reach a verdict.
Confidence level = 1 − alpha
Confidence level and significance level are two sides of the same coin. The significance level, called alpha, is the false-positive risk you're willing to tolerate. The confidence level is simply what's left over:
95% confidence ⟺ alpha = 0.05 ⟺ p-value < 0.05 to declare significance
So "95% confidence," "0.05 significance level," and "a 5% false-positive tolerance" all describe the same threshold. When a calculator reports a p-value below 0.05, your result has cleared 95% confidence. They are the same decision expressed in different vocabulary.
How alpha maps to confidence
| Alpha (significance) | Confidence level | False-positive tolerance |
|---|---|---|
| 0.10 | 90% | 1 in 10 |
| 0.05 | 95% | 1 in 20 |
| 0.01 | 99% | 1 in 100 |
Confidence level vs. confidence interval
The confidence level (e.g. 95%) is the reliability you choose. The confidence interval is what that choice produces: a range of plausible values for the true effect. A 95% confidence level yields a 95% confidence interval: the band that, across repeated tests, would contain the real difference about 95% of the time.
This is why the interval is more honest than a single lift number. If your test shows variant B is "+1.5%" with a 95% interval of +0.4% to +2.6%, the whole range sits above zero, so you have a credible winner. But if the interval reads −0.3% to +3.3%, it crosses zero. That means "no difference" is still on the table, and you should not call it.
Reading the interval in practice
- Interval entirely above zero: the variant is a winner at your chosen confidence.
- Interval crosses zero: inconclusive. You need more data or a bigger true effect.
- Width of the interval: narrower means more precise. Wide intervals usually signal a small sample.
The misreading that trips everyone up
The single most common mistake is reading "95% confidence" as "there's a 95% chance variant B is the better one." It does not mean that. The 95% describes the reliability of the testing procedure over the long run, not the probability that this particular result is correct.
Two consequences follow that matter for real decisions:
- A 95% result can still be wrong. Roughly 1 in 20 "significant" wins at this level are false positives. Confidence quantifies the risk; it doesn't erase it.
- Confidence says nothing about size. A result can clear 95% confidence and still represent a lift too small to be worth shipping. Always pair confidence with the effect size and the confidence interval before deciding.
Which confidence level to use
A 95% confidence level is the common default for marketing A/B tests because it balances false positives against how long you have to wait for a verdict. Adjust it to the stakes of the decision rather than treating it as fixed.
A practical rule of thumb
- 90%: fast, low-risk, easily reversible tests (a button color, an email subject line) where you'd rather learn quickly.
- 95%: the standard default for most landing-page and funnel experiments.
- 99%: high-stakes, hard-to-reverse changes such as pricing, checkout flow, or anything tied to revenue, where a wrong call is expensive.
Whatever you pick, set it before the test starts. Raising the bar after seeing the data, or stopping the moment a result crosses your line, quietly inflates your real false-positive rate well past the level you think you're holding.
Frequently asked questions
Does 95% confidence mean there's a 95% chance my variant is the winner?
No. That's the most common misreading. 95% confidence describes the long-run reliability of the method: if nothing changed, you'd wrongly declare a winner only about 5% of the time. It is not the probability that this specific variant is better.
What's the difference between confidence level and confidence interval?
The confidence level (e.g. 95%) is the reliability you choose up front. The confidence interval is the resulting range of plausible values for the true effect. A 95% level produces a 95% interval. If that range crosses zero, you don't have a winner.
Should I always use 95% confidence?
Not always. 95% is the common default, balancing false positives against speed. Drop to 90% for low-risk, reversible tests where you want answers faster, and raise to 99% for high-stakes changes like pricing or checkout where a wrong call is costly.
How does confidence relate to the p-value?
They are mirror images. Confidence level equals 1 minus alpha, and a result is significant when the p-value falls below alpha. So a p-value under 0.05 means you've cleared 95% confidence: the same decision, expressed two different ways.
Last updated: 14 June 2026