What Is a P-Value?
What a p-value actually measures
A p-value is a conditional probability. It answers one precise question: assuming the two variants are truly identical, how likely is it that random chance alone would produce a difference at least as large as the one you observed? The starting assumption, "there is no real difference," is called the null hypothesis.
A small p-value means your data would be surprising if the null were true, which is evidence against the null. A large p-value means your data is perfectly consistent with pure chance, so you have no reason to reject the idea that the variants perform the same. The p-value runs from 0 to 1.
Where the number comes from
In an A/B test, a statistical test (commonly a two-proportion z-test) converts your two conversion rates and sample sizes into a test statistic, and the p-value is the tail probability associated with that statistic. You don't compute it by intuition. A calculator or stats library does it for you.
Why 0.05 means significant at 95%
Before running a test you pick a significance threshold (alpha), and 0.05 is the long-standing convention. If your measured p-value lands below that threshold, you call the result statistically significant; if it lands above, you don't. The choice of 0.05 is a convention, not a law of nature.
The threshold and the confidence level are mirror images: confidence equals 1 − alpha. So a p-value below 0.05 corresponds to 95% confidence, below 0.01 to 99% confidence, and below 0.10 to 90% confidence. Alpha is also your accepted false-positive rate. At 0.05, you're agreeing to be fooled by chance about 1 time in 20.
| P-value threshold (alpha) | Confidence level | Typical use |
|---|---|---|
| 0.10 | 90% | Low-risk tests where speed matters |
| 0.05 | 95% | Default for most marketing A/B tests |
| 0.01 | 99% | High-stakes changes (pricing, checkout) |
What a p-value does and doesn't tell you
The p-value is widely misread, so it's worth being blunt about its limits. It tells you whether a difference is likely real. It says nothing about how big or how valuable that difference is.
What it does tell you
- How surprising your data is under the assumption of no effect.
- Whether to reject the null at your chosen threshold: a yes/no signal once you've set alpha.
What it does not tell you
- The probability your hypothesis is true. The p-value assumes the null is true; it can't also be the chance the null is false. A p of 0.03 is not "a 97% chance the variant wins."
- The size of the effect. With enough traffic, a trivial 0.1% lift can produce a very small p-value. Always read the lift and confidence interval beside it.
- That the result will replicate. Even a clean p-value below 0.05 carries a real false-positive risk, which is why peeking and stopping early are dangerous.
A quick example
Suppose variant A converts 100 of 2,000 visitors (5.0%) and variant B converts 130 of 2,000 (6.5%). A two-proportion z-test on those numbers returns a p-value of roughly 0.04.
Read that as: if A and B were genuinely identical, you'd expect a gap this large or larger about 4% of the time from chance alone. Because 0.04 is below the 0.05 threshold, the result is significant at 95% confidence and you can reasonably treat B as the winner, while noting the margin is thin, so more data would make it sturdier.
Frequently asked questions
What does a p-value of 0.05 mean?
A p-value of 0.05 means that if the two variants were actually identical, you'd see a difference at least this large about 5% of the time by random chance. It's the conventional cutoff for calling a result statistically significant at 95% confidence.
Is a smaller p-value a bigger effect?
No. A smaller p-value means more evidence that the difference isn't chance, not that the difference is larger. A tiny lift can produce a very small p-value with enough traffic. Always read the effect size and confidence interval alongside the p-value.
Does a p-value tell you the probability the variant is better?
No, and this is the most common misreading. The p-value assumes the null hypothesis is true and measures how surprising your data is under it. It is not the probability that your hypothesis is correct or that the variant truly wins.
What's the difference between a p-value and confidence?
They're two sides of the same result. Confidence level equals 1 minus your significance threshold: a p-value under 0.05 corresponds to 95% confidence. The p-value is the measured probability; the confidence level is the threshold you set in advance to judge it against.
Last updated: 14 June 2026