A/B Test Significance Calculator: How to Use It

By Amit Gupta · Senior Director, Marketing Operations · Published 14 June 2026

What to enter for each variant

Enter two numbers per variant: Visitors (everyone who could have converted) and Conversions (how many actually did). The calculator computes each conversion rate for you, so you never type a percentage. It needs the raw counts because the sample size is what makes a result trustworthy.

Define "visitor" and "conversion" consistently

Use the same denominator on both sides. If variant A counts unique sessions, variant B must too. Mixing sessions with users quietly biases the test. A "conversion" is your one chosen success event: a signup, a purchase, a demo request. Pick one per test and keep it fixed.

Keep the two variants comparable

The math assumes both groups were drawn from the same traffic at the same time, split randomly. If one variant ran on a different week, audience, or device mix, the z-test will still produce a number, but the difference may reflect those conditions, not your change.

How to read z-score, p-value, and confidence

These three numbers are the same evidence expressed three ways. The z-score measures how many standard errors apart the two rates are; the p-value is the chance of seeing a gap that large if the variants were truly identical; and confidence is what the calculator reports as 1 minus the p-value, shown as a percentage.

The thresholds to remember

• A z-score around 1.96 equals 95% confidence (a p-value of about 0.05), the usual line for declaring a winner.
• A z-score of 2.58 equals 99% confidence (p ≈ 0.01), a stricter bar for high-stakes changes like pricing.
• Bigger z, smaller p, higher confidence all mean the same thing: the gap is less likely to be random luck.

Confidence answers "is this difference real?", not "is it big enough to matter?" Always read it next to the relative lift the calculator shows, so a statistically real but trivial change doesn't get shipped on its own.

What the ship-it verdict means

The verdict translates the confidence level into a decision so you don't have to memorise thresholds. It maps directly to the confidence percentage and is colour-coded in the result card.

"Ship it" is a green light on the statistics, not a blanket recommendation. Pair it with the relative lift and your own judgement on whether the effect is worth the engineering cost. A significant +0.1% rarely earns a rollout; a significant +8% usually does.

How to use the sample-size hint

Whenever a test falls short of 95%, the calculator shows a "Sample to reach 95%" figure: roughly how many visitors per variant you'd need to confirm an effect the size you're currently seeing. Treat it as a planning estimate, not a promise.

Read it as "how much further to go"

Compare the suggested per-variant sample to the traffic you already have. If you're at 5,000 per variant and it suggests 18,000, you're roughly a third of the way and should keep the test live. If the number is enormous, your true effect is probably tiny, which is itself a useful answer.

Set the size before you launch, not after

The honest way to use the hint is to estimate it up front, commit to that sample size, and run the test to completion before reading the verdict. Deciding the finish line in advance is what protects you from peeking, the most common way A/B tests produce false winners.

A full worked walkthrough

Use the calculator's default scenario to see every output line up. Variant A has 5,200 visitors and 208 conversions; variant B has 5,180 visitors and 245 conversions.

1. Enter the counts. Type 5,200 and 208 for A, then 5,180 and 245 for B. The rates work out to about 4.0% for A and 4.7% for B.
2. Read the confidence. The two-tailed z-test turns that gap into roughly 93% confidence: past 90% but short of the 95% line, so the verdict reads Strong signal. That is real enough to take seriously, not yet a clear Ship it.
3. Check the relative lift. B sits about 18% above A in relative terms, a lift big enough to be worth confirming, even though the test hasn't yet cleared the 95% bar.
4. Use the sample hint. Because confidence is under 95%, the "Sample to reach 95%" figure appears (here on the order of ten-thousand-plus visitors per variant), telling you roughly how much more traffic would settle it. Run to that size, then read the verdict once.

Change one input and watch the whole card move: add a few conversions to B and confidence climbs past 95%, flipping the verdict to Ship it. Drop them and it slides toward Inconclusive.

Last updated: 14 June 2026