Incrementality calculator: did your ads cause these sales?
Incrementality asks one question: did your ads cause these sales, or would they have happened anyway? To answer it, keep ads away from a random holdout group and compare how often each group buys. Drag each group's bar to its buying rate, or type your test's numbers, to see the sales the ads caused, the lift with its confidence range, and iROAS, the return on spend from caused sales only.
Did your ads cause these sales, or would they have happened anyway?
Keep ads away from a random group of people, the holdout, and compare how often each group buys. The holdout’s rate is what would have happened anyway; anything above it, the ads caused.
Your result
Significant liftPeople who could see the ads bought 25.0% more often than the holdout: about 600 sales the ads caused. Each $1 of spend caused $1.20 of revenue.
Drag the top of either bar to change its conversion rate, and the divider below to change how many people you hold out.
How often each group bought
Lift, with its 95% confidence range
Lift is how much more often the ad group bought than the holdout. The bar is the range the true lift very likely sits in; if it crosses “No effect”, the ads may have done nothing.Holdout share
The share of people kept from seeing ads. Drag the divider: a bigger holdout narrows the range but gives up the sales its people would have made from ads.The platform credited 2,400 orders; the ads caused 25% of that many. Multiply the platform’s ROAS by that share to estimate iROAS between tests.
- Sales the ads causedIncremental conversions: orders from the ad group above what the holdout's buying rate predicts. The range is the 95% confidence range.
- 600 (likely 381 to 819)
- LiftHow much more often the ad group bought than the holdout, as a share of the holdout's rate. 1.5% against 1.2% is a 25% lift.
- +25.0%
- Confidence range for liftThe 95% confidence range: for a test this size, the true lift is very likely inside it. If it includes 0%, the ads may have done nothing.
- +16% to +34%
- Sure it isn't chanceConfidence from a two-proportion z-test: how unlikely a gap this big is if the ads did nothing. 95% or more is the usual bar.
- 100.0% · p < 0.0001
- Holdout shareThe share of people kept from seeing ads. A bigger holdout narrows the range but gives up the sales those people would have made from ads. 10–20% is common.
- 20%
- Share of ad-group sales causedSales the ads caused ÷ every order from the people who could see ads. The rest would have happened anyway.
- 20.0%
- Cost per sale causedAd spend ÷ sales the ads caused: the real cost of each extra order.
- $66.67
- iROASIncremental return on ad spend: revenue from the sales the ads caused ÷ ad spend. Compare it with your break-even ROAS, 1 ÷ gross margin: 2x at a 50% margin.
- 1.20x
- Caused ÷ platform-creditedThe incrementality factor: how many of the orders the platform credited the ads really caused. Multiply platform ROAS by it to estimate iROAS between tests.
- 25%
Find trends in your business for free. Ask a question about your store and Replo answers it from every tool you connect.
How to read your number
Incrementality is the share of your conversions that happened because of an ad, measured against a holdout group that never saw it. The holdout converted at 1.20%, so that's what the test group would have done without ads. It converted at 1.50% instead: a 25% lift, or 600 of its 3,000 conversions. On $40,000 of spend and $80 per conversion, that's $48,000 of incremental revenue, an iROAS of 1.20x, even though the platform credited 2,400 conversions.
Why it matters
Platform ROAS counts every buyer who saw or clicked an ad, including people who would have bought anyway. Incrementality counts only the sales the ads caused. A campaign can show a 4x ROAS and a 1x iROAS; the second number is the one that says whether to keep paying for it.
Formula
Lift = (test conversion rate − holdout conversion rate) ÷ holdout conversion rate
Incremental conversions = test conversions − (holdout conversion rate × test group size). iROAS = incremental conversions × revenue per conversion ÷ ad spend. The significance check is a two-sided, two-proportion z-test at 95% confidence.
New to the term? Read what Marketing attribution means in the glossary.
Other ways to calculate incrementality
- Incremental conversions
- Incremental conversions = test conversions − holdout rate × test group size
- The count of sales the ads caused. Use it for cost per incremental conversion: spend ÷ incremental conversions.
- iROAS (incremental ROAS)
- iROAS = incremental revenue ÷ ad spend
- When deciding whether a channel pays for itself. Compare it with your break-even ROAS, not with the platform's ROAS.
- Incrementality factor
- Incrementality factor = incremental conversions ÷ conversions the platform credited
- Between tests: multiply the platform's reported ROAS by this share to estimate iROAS until you can test again.
- Geo lift
- Lift = (test regions' sales − predicted sales without ads) ÷ predicted sales
- For a geo holdout, where the prediction comes from how the test regions tracked the control regions before the test, not from one conversion rate.
What is incrementality?
Incrementality is the share of your results that happened because of an ad, not just alongside it. If 3,000 people who saw your ads bought, but 2,400 of them would have bought anyway, only 600 sales were incremental.
You measure it with a holdout test: keep ads away from a random group, then compare how often each group converts. The holdout's rate is your baseline, and the difference is the lift your ads caused.
Geo holdout vs audience holdout
There are two common ways to hold people out, and they need different math.
Audience holdout (Meta's Conversion Lift, Google's Conversion Lift): the platform splits people at random and never shows ads to the holdout. Groups are large and comparable, so this calculator's z-test fits. You need the platform to run it, and it only measures that platform.
Geo holdout: you turn ads off in some regions, such as states or cities, and keep them on in others. It works for any channel, including TV, podcasts and creators, and you read the result in your own store's sales. With only a few dozen regions, though, the regions differ more than people do, so analyse it against a forecast built from the weeks before the test, as in Google's geo experiments method, rather than with a conversion-rate test alone.
How to calculate incrementality: a worked example
A store runs a Meta holdout for four weeks. 200,000 people could see the ads and 3,000 of them bought; 50,000 were held out and 600 of them bought. The test group's ads cost $40,000, an order is worth $80, and Meta credited 2,400 conversions.
- Holdout rate = 600 ÷ 50,000 = 1.20%; test rate = 3,000 ÷ 200,000 = 1.50%
- Lift = (1.50% − 1.20%) ÷ 1.20% = 25%
- Incremental conversions = 3,000 − 1.20% × 200,000 = 600 (95% range 381 to 819)
- iROAS = 600 × $80 ÷ $40,000 = 1.20x; cost per incremental conversion = $66.67
- Incremental ÷ credited = 600 ÷ 2,400 = 25%
The lift is significant (z = 5.0), but each $1 of spend caused only $1.20 of revenue. At a 50% margin that loses money on the first order, so the campaign has to earn it back through repeat purchases.
What is a good incremental lift?
There's no universal number, because lift depends on how many people buy without ads. A brand most people already know will see a small lift; a new product with little word of mouth can see a large one. What matters is whether the iROAS clears your break-even ROAS (1 ÷ gross margin).
Judge a test on its range, not its single figure. If the 95% range of incremental conversions includes zero, you haven't shown the ads did anything; if its low end still clears break-even, you can scale with confidence.
| Audience holdout | Geo holdout | |
|---|---|---|
| Who's held out | Random people, chosen by the ad platform | Whole regions you choose |
| Channels it measures | The platform running it | Any channel, including offline |
| Where you read results | The platform's lift report | Your own store's sales |
| Typical analysis | Two-proportion test on conversion rates | Forecast from a pre-test period |
| Main risk | Only measures conversions the platform can see | Regions differ; needs enough of them and enough weeks |
How to run a holdout test
- Pick one question. Does this campaign, or this channel, cause sales? Test one thing at a time.
- Size the holdout first. A small holdout costs fewer sales but takes longer to reach significance; plan it with your baseline rate before you start.
- Run whole weeks. Weekday and weekend buying both count, and the test should outlast your usual time from first visit to order.
- Change nothing mid-test. New creative or budget shifts change what the test measures.
- Use the result between tests. Turn it into an incrementality factor and apply it to the platform's daily ROAS until you test again.
Questions about incrementality
- What does incrementality mean?
- Incrementality means the sales your marketing caused, as opposed to sales that would have happened anyway. If 3,000 people who saw your ads bought and 2,400 of them would have bought without ads, the incremental sales are 600.
- What is incrementality testing?
- It's a controlled experiment that measures how many sales an ad caused. One group can see the ads and a random holdout can't; the difference in their conversion rates is the lift the ads caused, beyond what would have happened anyway.
- How do you calculate incremental lift?
- Subtract the holdout's conversion rate from the test group's, then divide by the holdout's rate. A test group at 1.5% against a holdout at 1.2% is a 25% lift.
- What is iROAS?
- Incremental return on ad spend: the revenue the ads caused, divided by what they cost. It's usually lower than platform ROAS, because the platform also counts buyers who would have ordered without the ad.
- How big should a holdout group be?
- Big enough that the expected gap is detectable. With a 1% to 2% conversion rate, tens of thousands of people per group is typical. Use the sample size planner with your baseline rate and the smallest lift worth finding.
- Is a geo holdout better than a platform lift test?
- Neither is better everywhere. A platform lift test is cleaner because people are randomised, but it only measures that platform. A geo test works for any channel and reads your real sales, but needs more regions, more weeks and a forecast to compare against.
Sources
Experiments
A/B test your pages in Replo
Split traffic between two versions of a Replo page and see which one sells, with the result next to your sales.
