What Is the Peeking Problem in A/B Testing?
What peeking means
Peeking is the habit of watching a live A/B test, glancing at the significance number day after day, and calling a winner the first time it crosses the threshold, usually a p-value below 0.05, or "95% significant" in a dashboard. The mistake is not the looking; it is letting what you see decide when to stop.
It happens because classical significance testing assumes a single, pre-planned check at a fixed sample size. Stakeholders, though, want to know "is it working yet?", so they refresh the report, see green, and ship. That looks like diligence but quietly breaks the statistics the test relies on.
The pattern that causes the harm
The damaging version is: run the test, check it every day, and end it the instant significance appears. Each peek is a fresh opportunity for random noise to push the result over the line. The more often you look, and the more eagerly you stop, the more often you will declare a winner that does not exist.
Why it inflates false positives
It inflates false positives because a 5% significance level budgets for exactly one test, and conversion data wanders. Even when two variants are truly identical, the measured difference drifts up and down as visitors arrive, and at some point along that random walk the p-value will dip under 0.05 purely by chance. Peeking lets you catch that fleeting dip and freeze it as a "result."
Think of significance as a 5% chance of a false alarm per look. One look, one 5% risk, fine. But twenty looks give twenty chances for noise to trip the alarm, and those chances compound. The probability that at least one peek shows a spurious win grows steadily toward certainty the longer you keep checking.
This is the same logic as the multiple-comparisons problem: testing one hypothesis many times is statistically similar to testing many hypotheses once. Either way, you have given randomness more rolls of the dice, so the chance of a fluke passing your bar is no longer the 5% the number implies.
How bad does it get?
It can get bad enough that "significant" stops meaning much. With a single planned check, your false-positive rate is the 5% you signed up for. With continuous monitoring and a stop-on-significance rule, simulations of A/A tests (where both variants are identical) routinely show the spurious-win rate climbing into the tens of percent rather than staying at 5%.
The exact figure depends on how often you peek, how early you start, and how long you let the test run, so treat any single number with caution. The reliable rule of thumb is directional: the more frequently you check and the more willing you are to stop early, the higher your effective false-positive rate climbs above 5%.
What it looks like in practice
The classic symptom is the "winner that didn't win": a variant declared a 10% lift after three days, shipped, and then no lift ever materializes in the actual numbers. Run enough peeked tests and you accumulate a backlog of these phantom wins: a roadmap built on noise, and a team that slowly loses trust in experimentation.
How to fix the peeking problem
You fix it by deciding how you will stop before the test starts, instead of letting the live result decide for you. There are three established approaches, in rough order of simplicity.
- Fixed-horizon testing. Calculate the required sample size up front from your baseline rate, the minimum effect you care about, and your desired power. Commit to running until you hit it, and check significance only once, at the end. It is the simplest discipline and the easiest to govern.
- Sequential testing. If you really need to look early, use a method built for it: group sequential designs with alpha-spending boundaries, or always-valid sequential tests. These adjust the significance threshold to account for repeated looks, so monitoring no longer inflates your error rate. Many modern experimentation platforms offer this mode.
- Bayesian testing. Report the probability that B beats A, or the expected loss of choosing wrong, instead of a fixed-error p-value. Bayesian results hold up better under continuous monitoring, though you still need sensible stopping rules and enough data to avoid acting on an unstable early estimate.
The non-negotiable habit: separate looking from deciding. Monitor freely for bugs, tracking failures, or serious harm, but only declare a winner using the stopping rule you chose in advance. If you used a plain fixed-horizon design, that means waiting for the planned sample size and reading the result exactly once.
Frequently asked questions
Why does peeking inflate the false positive rate?
A standard 5% significance level assumes you test once. Conversion data wanders randomly, so the p-value dips below 0.05 by chance at some point in almost any test. If you check repeatedly and stop the moment it does, you give yourself many chances to find that fluke, so your real false positive rate climbs far past 5%.
Is it ever okay to look at an A/B test early?
Looking is fine; stopping on what you see is the problem. You can monitor a test for bugs, tracking errors, or severe harm at any time. What you must not do is end the test and declare a winner just because the p-value crossed 0.05 before you reached your planned sample size.
How is the peeking problem different from p-hacking?
They are cousins. P-hacking is any practice that manufactures significance: trying many metrics, segments, or stopping rules until something passes. The peeking problem is one specific form: repeatedly testing the same metric over time and stopping at the first significant reading. Both inflate false positives.
Does Bayesian testing remove the peeking problem entirely?
It reduces it but does not magically erase it. Bayesian methods report a probability of being better rather than a fixed-error p-value, so continuous monitoring is less catastrophic. But stopping the instant a number looks good still biases your estimate, so sensible stopping rules and adequate data still matter.
Last updated: 14 June 2026