How Big Should My A/B Test Sample Be?
What decides the sample size
Four inputs set your required sample size. Change any one and the number moves, sometimes a lot. Get these right before you write a line of test code.
Baseline conversion rate
This is how the control variant already performs. Lower baselines need more traffic: distinguishing a real lift on a 1% conversion page is much harder than on a 20% page, because there are fewer conversions to learn from per visitor.
Minimum detectable effect (MDE)
The MDE is the smallest lift you care about catching, say "a relative +5% on the baseline." This is the single biggest lever. The smaller the effect you insist on detecting, the larger the sample required, and the relationship is steep, not linear.
Significance level
Significance (alpha) is your tolerance for a false positive: calling a flat test a winner. The common default is 95% confidence, i.e. a 5% false-positive rate. Tightening to 99% raises the sample because you are demanding stronger evidence.
Statistical power
Power is the chance of detecting a real effect that genuinely exists, so you avoid false negatives. A common default is 80% power. Raising it to 90% catches more true winners but, again, costs more visitors.
How to calculate it, step by step
You do not need to do the algebra by hand, but you do need to choose the inputs deliberately. Follow this order.
- Measure your baseline. Pull the control's current conversion rate from real, recent data over a full business cycle (weekday and weekend behaviour differ).
- Decide the minimum lift worth shipping. Pick the smallest relative improvement that would justify the change. A +0.2% lift you will never notice is not worth a giant sample.
- Set significance and power. Default to 95% significance and 80% power unless the decision is high-stakes (then consider 99% / 90%).
- Run the numbers. Feed those four inputs into a sample-size calculator to get visitors-per-variant.
- Translate to a timeline. Divide required visitors per variant by your daily traffic per variant to estimate test duration, and round up to whole weeks.
- Commit to the number. Write it down before launch and run the test to that size, with no stopping early.
Rough guidance and ranges
Exact figures come from the calculator, but a few rules of thumb help you sanity-check the answer before you start.
- Halving the effect roughly quadruples the sample. Precision scales with the square root of the count, so chasing tiny lifts gets expensive fast.
- Lower baselines cost more traffic. A 2% page generally needs far more visitors than a 30% page to detect the same relative lift.
- Most real tests on low-converting pages land in the tens of thousands of visitors per variant to detect a modest relative lift at 95% / 80%. Treat that as a ballpark, not a target.
How the four inputs push sample size:
| Input | Move it this way | Required sample |
|---|---|---|
| Minimum detectable effect | Smaller lift to detect | Goes up sharply |
| Baseline conversion rate | Lower baseline | Goes up |
| Significance level | 95% → 99% | Goes up |
| Statistical power | 80% → 90% | Goes up |
The peeking pitfall
The most common way teams ruin a correctly sized test is by reading it too early. Watching the dashboard and stopping the moment it crosses 95% (call it "peeking") quietly inflates your false-positive rate far above the 5% you signed up for, because every extra look is another chance for noise to fake a win.
The fix is simple: calculate the sample size up front, run the test until you hit it (and at least one full business cycle), and read the result once. If you genuinely need to monitor mid-flight, use a method designed for it, such as sequential testing or a pre-planned interim analysis, not casual peeking at a fixed-horizon test.
Frequently asked questions
What is a good sample size for an A/B test?
There is no universal number. Sample size is driven by your baseline conversion rate and the smallest lift you want to detect. As a rough feel, detecting a relative lift of a few percent on a low-converting page often needs tens of thousands of visitors per variant.
Why does a smaller effect need a bigger sample?
Small differences are easily drowned out by random noise. To distinguish a tiny real lift from chance, you need many observations so the noise averages out. Detecting half the effect roughly quadruples the required sample, because precision scales with the square root of the count.
Can I stop the test early once I reach my sample size?
Stop only once you have reached the sample size you calculated up front and run a full business cycle. Stopping the moment a test looks significant (peeking) inflates false positives well beyond your stated 5%. Fix the sample, then read the result once.
How do significance and power change the sample size?
Higher significance (95% to 99%) and higher power (80% to 90%) both demand more visitors because you are asking for stronger evidence and a smaller chance of missing a real effect. Most teams default to 95% significance and 80% power as a practical balance.
Last updated: 14 June 2026