Conversion optimization
Shopify A/B testing best practices
How to A/B test a Shopify store the right way: Shopify's built-in Rollouts experiment, the best practices that keep results honest and what to test first.
Start for freeShopify A/B testing best practices start with changing one thing and judging it by orders or revenue per visitor. Size the test before you launch, then run it for at least two full weeks. Shopify can test theme changes with a rollout set to Experiment, on the Grow plan or higher. A/B testing in Replo tests the landing pages and product pages Replo builds for your store.
Etsy's engineers learned why a test should check one idea at a time. They built infinite scroll for Etsy's search results, and shoppers who got it saw fewer items, clicked fewer and bought fewer items from search. The team then tested each assumption on its own. More items per page didn't raise purchases, and slower pages made no measurable difference (Dan McKinley's write-up).

Can you A/B test on Shopify without an app?
Yes, for changes to your theme. Shopify's Rollouts can run an Experiment. Half your visitors see your theme as it is, and half see it with your changes.

- In your Shopify admin, go to Markets > Rollouts and click Create rollout.
- Name it and pick Experiment as the type.
- Click Add changes, then Edit theme, and make your change in the theme editor.
- Keep the 50/50 split or change it, then set the dates. An Experiment ends after 90 days unless you pick another date.
Rollouts test your main theme, your checkout setup, catalogs and discounts. Pages outside your theme, like a landing page from a page builder, need that tool's own A/B testing.
What are the A/B testing best practices for Shopify?
- Test the page that gets your ad traffic. Start where your ads land, or with the product page that sells most.
- Write one guess you can prove wrong. For example: "If we put the review count above the headline, more visitors will add to cart."
- Size the test before you launch. Work out how many visitors each version needs.
- Split traffic, then wait. Don't stop when one version pulls ahead early.
- Ship the winner and log the result. Write down what you learned, then test the next idea.
Etsy's lesson fits step 2. Test the smallest version of an idea that can prove your guess wrong, before you build the whole feature.

How many visitors does a Shopify A/B test need?
The number depends on your conversion rate and the lift you want to catch. A page converting at 3% needs about 14,000 visitors per version to catch a 20% lift. At 1,000 visitors a day, that takes about four weeks.

Size yours with the A/B test sample size calculator. A result counts when the new version has a 95% chance of really being better. Check it with the statistical significance calculator.
Which metrics should a Shopify A/B test use?
Pick one metric before you start:
- Conversion rate: the share of visitors who buy. Use it when the change is about getting the sale.
- Add-to-cart rate: it moves sooner than orders, so it suits stores with less traffic.
- Revenue per visitor: sales divided by visitors. Use it when the change touches price, bundles or the offer, because it counts order size too.

What should you A/B test on a Shopify store?
Test the parts near the top of the page first, where every visitor looks:
- The headline: a benefit, or the product's name.
- The first product photo: the product alone, or the product with its benefit.
- The button: its words first, then its color and size.
- Reviews: above the headline, or below the offer.
- The offer: one bundle price, or a pack size picker.

Our A/B testing examples for Shopify show each idea on two real store pages.
Can you run multivariate tests on Shopify?
You can, but most stores don't have the traffic. A multivariate test changes several parts at once and tests every combination.
Four headlines and three hero photos make 12 versions, and each one needs its own sample. Our guide on how to A/B test a landing page shows a real 24-version test. Most stores learn faster from one A/B test after another.
What are the best A/B testing tools for Shopify?
Pick the tool by where the page lives:
- Theme pages: Shopify's Rollouts runs an Experiment on your theme, on the Grow plan and up. Replo Theme Builder has its own A/B testing for theme pages.
- Landing pages and product pages built in Replo: A/B testing in Replo splits visitors on one link, with results next to your orders.
What mistakes should you avoid?
- Too many changes at once. You'll get a winner but not the reason.
- Stopping early. Early leads often fade. Run full weeks until you reach your sample size.
- Watching clicks, not revenue. A discount can lift conversion rate and cut average order value.
- Skipping phones. Most store traffic comes from phones. Check the new version there, and read results by device.
- Two tests on one page. The results mix together. Run them one after the other.
How do you A/B test landing pages with Replo?
A/B testing in Replo runs on the pages Replo builds and hosts on your domain. Ask in chat, for example "Test reviews above the headline on my product page". Replo copies the page and makes the change on the copy. It then splits visitors on one link before the page loads, so nothing flickers.
Replo calls a winner only at a 95% chance to beat the original. Every version first needs 100 visitors, 10 conversions and a full week.

Read the full walkthrough in how to A/B test landing pages, or start for free.
Frequently asked questions
- How long should a Shopify A/B test run?
- Run it for at least two full weeks, so weekday and weekend shoppers both count. Then keep going until each version reaches the sample size you planned. Most stores land on two to four weeks for a clean result.
- Can I split test Shopify checkout pages without Shopify Plus?
- Only in a limited way. Shopify restricts checkout customization outside Shopify Plus, so there is little to change on the checkout itself. Test the product page, the cart and the offer instead, since they decide who reaches checkout.
- What traffic volume is enough for ab testing retail audiences?
- Enough for each version to reach its planned sample within a few weeks. A page converting at 3% needs about 14,000 visitors per version to catch a 20% lift. At 1,000 visitors a day, that takes about four weeks.
- Does split testing Shopify pages slow down my site speed?
- It can, if a tool swaps content after the page loads. That adds a script and can flash the original page first. Replo splits visitors before the page loads, so nothing flickers. With any setup, check page speed on a phone while the test runs.



