Statistical significance calculator for A/B tests
Enter each version's visitors and conversions to see whether the gap is real. Then plan your next test's sample size and length.
Formula
z = (p₂ − p₁) ÷ √(p(1 − p)(1/n₁ + 1/n₂))
This is a two-tailed, two-proportion z-test. p₁ and p₂ are the two conversion rates, n₁ and n₂ the visitors, and p the pooled rate. The p-value comes from the normal distribution; the 95% interval uses each version's own rate. Sample sizes use the standard two-proportion formula at 95% significance and 80% power.
New to the term? Read what A/B testing means in the glossary.
How to read the result
- Confidence is 1 minus the p-value. At 95% or above, the calculator calls the result significant.
- The p-value is how often you'd see a gap at least this big if the two versions truly converted the same. Below 0.05 is the usual bar.
- The 95% interval is the range the real gap in conversion rate probably sits in. If it runs from just above zero to well above it, the variant likely wins, but maybe by less than the headline lift.
A significant result tells you the gap is probably real. It doesn't tell you the gap is big enough to matter, so read the interval as well as the verdict.
How to calculate statistical significance: a worked example
A store tests its product name as the headline (control) against the hook from its best ad (variant), with a 50/50 split.
- Control: 5,000 visitors, 150 orders, a 3.0% conversion rate
- Variant: 5,000 visitors, 190 orders, a 3.8% conversion rate
- The variant converts 26.7% better; p = 0.027 and z = 2.21
- The interval for the gap runs from +0.09 to +1.51 points
The variant very likely wins, but the true lift could be much smaller than 26.7%. With 170 orders on the variant instead (3.4%), p = 0.26: that 13% lift could easily be chance.
How many visitors a test needs
The smaller the lift you want to detect, the more visitors you need. At a 3% conversion rate, finding a 10% lift takes 53,211 visitors per version; finding a 20% lift takes 13,914.
At 2,000 visitors a day split across two versions, that's about 54 days or 14 days. The table gives more starting points, all at 95% significance and 80% power.
| Current conversion rate | To find a 10% lift | 20% lift | 30% lift |
|---|---|---|---|
| 1% | 163,095 | 42,693 | 19,827 |
| 2% | 80,682 | 21,109 | 9,798 |
| 3% | 53,211 | 13,914 | 6,455 |
| 5% | 31,234 | 8,158 | 3,780 |
Five ways to fake a winner
- Stopping when it first hits 95%. Confidence swings early. Decide the sample size first and stop when you reach it.
- Running less than a week. Weekend shoppers behave differently. Run whole weeks.
- An uneven split. If a 50/50 test ends at 5,400 vs 4,600 visitors, something in the setup is broken. Fix it before you trust the result.
- Too many versions. Each extra version is another chance of a false winner. Test two, or raise the bar.
- Changing the page mid-test. Any edit restarts the clock.
Run the test where you build the page
A/B testing on Replo sites splits traffic between two versions of a page; check the result here. Theme Builder, Replo's Shopify page builder, has A/B testing built in for pages in your theme today. See A/B testing with Replo for how it works.
Questions about statistical significance
- What does a statistical significance calculator do?
- It tells you whether the gap between two conversion rates is big enough that chance is an unlikely explanation. You enter each version's visitors and conversions; it returns a p-value, a confidence level and a verdict at your threshold, usually 95%.
- What p-value counts as significant?
- Below 0.05, which this calculator shows as 95% confidence or more. For a change that's costly to undo, such as a price, use a stricter bar like 0.01, and plan a bigger sample to match.
- How long should an A/B test run?
- Until each version reaches the sample size you planned, and for at least one full week, so weekdays and weekends both count. At low traffic, test bigger changes: a 20% lift needs about a quarter of the visitors a 10% lift does.
- Can I stop my test as soon as it reaches 95%?
- No. Checking every day and stopping at the first 95% inflates false winners. Decide the sample size before you start with the planner above, and call the test only when both versions reach it.
- Can I use this with the testing tool I already have?
- Yes. Paste the visitors and conversions from any tool, such as Intelligems, VWO or Shopify Rollouts, and the math is the same. Those tools still work with pages you build in Replo.
Experiments
A/B test your pages in Replo
Split traffic between two versions of a Replo page and see which one sells, with the result next to your sales.