Skip to main content
Blog

Conversion optimization

Shopify A/B testing best practices

How to A/B test a Shopify store the right way: Shopify's built-in Rollouts experiment, the best practices that keep results honest and what to test first.

Start for free

Shopify A/B testing best practices start with changing one thing and judging it by orders or revenue per visitor. Size the test before you launch, then run it for at least two full weeks. Shopify can test theme changes with a rollout set to Experiment, on the Grow plan or higher. A/B testing in Replo tests the landing pages and product pages Replo builds for your store.

Etsy's engineers learned why a test should check one idea at a time. They built infinite scroll for Etsy's search results, and shoppers who got it saw fewer items, clicked fewer and bought fewer items from search. The team then tested each assumption on its own. More items per page didn't raise purchases, and slower pages made no measurable difference (Dan McKinley's write-up).

Etsy engineers and customer service staff at desks in an open loft office, with team photos pinned to a low wall in front.
Etsy engineers at work in their office. After infinite scroll lost, Dan McKinley advised testing a minimal version first and planning on being wrong. Photo: Wikimedia Commons, CC BY-SA 2.0, by Charles & Hudson

Can you A/B test on Shopify without an app?

Yes, for changes to your theme. Shopify's Rollouts can run an Experiment. Half your visitors see your theme as it is, and half see it with your changes.

Two Shopify Help Center pages, key lines outlined. “Creating a rollout”: create one from the Markets > Rollouts page of your Shopify admin, then add changes and details. “Types of rollouts and changes”: use an Experiment to test changes against a control.
From Shopify's Help Center: an Experiment is the rollout type that tests changes against a control.
  1. In your Shopify admin, go to Markets > Rollouts and click Create rollout.
  2. Name it and pick Experiment as the type.
  3. Click Add changes, then Edit theme, and make your change in the theme editor.
  4. Keep the 50/50 split or change it, then set the dates. An Experiment ends after 90 days unless you pick another date.

Rollouts test your main theme, your checkout setup, catalogs and discounts. Pages outside your theme, like a landing page from a page builder, need that tool's own A/B testing.

What are the A/B testing best practices for Shopify?

  1. Test the page that gets your ad traffic. Start where your ads land, or with the product page that sells most.
  2. Write one guess you can prove wrong. For example: "If we put the review count above the headline, more visitors will add to cart."
  3. Size the test before you launch. Work out how many visitors each version needs.
  4. Split traffic, then wait. Don't stop when one version pulls ahead early.
  5. Ship the winner and log the result. Write down what you learned, then test the next idea.

Etsy's lesson fits step 2. Test the smallest version of an idea that can prove your guess wrong, before you build the whole feature.

Diagram: one link sends 50% of visitors to version A, the original page, and 50% to version B, the same page with one change to the headline. Both feed into orders, where you keep the version that sells.
Both versions share one link and the same visitors, so the one change is the only difference.

How many visitors does a Shopify A/B test need?

The number depends on your conversion rate and the lift you want to catch. A page converting at 3% needs about 14,000 visitors per version to catch a 20% lift. At 1,000 visitors a day, that takes about four weeks.

Replo's significance calculator: a page converting at 3% with 1,000 visitors a day, looking for a 20% lift, needs 13,914 visitors per version, about 28 days.
Put in your conversion rate, daily visitors and the lift you hope for. The answer is visitors per version.

Size yours with the A/B test sample size calculator. A result counts when the new version has a 95% chance of really being better. Check it with the statistical significance calculator.

Which metrics should a Shopify A/B test use?

Pick one metric before you start:

  • Conversion rate: the share of visitors who buy. Use it when the change is about getting the sale.
  • Add-to-cart rate: it moves sooner than orders, so it suits stores with less traffic.
  • Revenue per visitor: sales divided by visitors. Use it when the change touches price, bundles or the offer, because it counts order size too.
Replo's heatmap panel over a dog toy store's published page: revenue, carts and purchases for the page, and each element ranked by the revenue it brought in, led by a Shop Best Sellers button at $662.
Revenue per element shows which button earned the sales, so you know what to test next.

What should you A/B test on a Shopify store?

Test the parts near the top of the page first, where every visitor looks:

  • The headline: a benefit, or the product's name.
  • The first product photo: the product alone, or the product with its benefit.
  • The button: its words first, then its color and size.
  • Reviews: above the headline, or below the offer.
  • The offer: one bundle price, or a pack size picker.
Two iPhones. Left: Catalyst Pet's product page headline “One box. Everything your cat's routine needs.” over a cat photo. Right: Our Place's product page with a pot photo and the headline “Ceramic Nonstick Perfect Pot”.
Two real product pages make opposite headline choices: Catalyst Pet says why, Our Place says what.

Our A/B testing examples for Shopify show each idea on two real store pages.

Can you run multivariate tests on Shopify?

You can, but most stores don't have the traffic. A multivariate test changes several parts at once and tests every combination.

Four headlines and three hero photos make 12 versions, and each one needs its own sample. Our guide on how to A/B test a landing page shows a real 24-version test. Most stores learn faster from one A/B test after another.

What are the best A/B testing tools for Shopify?

Pick the tool by where the page lives:

  • Theme pages: Shopify's Rollouts runs an Experiment on your theme, on the Grow plan and up. Replo Theme Builder has its own A/B testing for theme pages.
  • Landing pages and product pages built in Replo: A/B testing in Replo splits visitors on one link, with results next to your orders.

What mistakes should you avoid?

  • Too many changes at once. You'll get a winner but not the reason.
  • Stopping early. Early leads often fade. Run full weeks until you reach your sample size.
  • Watching clicks, not revenue. A discount can lift conversion rate and cut average order value.
  • Skipping phones. Most store traffic comes from phones. Check the new version there, and read results by device.
  • Two tests on one page. The results mix together. Run them one after the other.

How do you A/B test landing pages with Replo?

A/B testing in Replo runs on the pages Replo builds and hosts on your domain. Ask in chat, for example "Test reviews above the headline on my product page". Replo copies the page and makes the change on the copy. It then splits visitors on one link before the page loads, so nothing flickers.

Replo calls a winner only at a 95% chance to beat the original. Every version first needs 100 visitors, 10 conversions and a full week.

Replo's A/B test results: a 25% off headline leads the original 8.5% to 7.5% on add to cart, an 81% chance to win, so the test keeps running until it reaches 95%.
The new headline is ahead, but at 81% Replo keeps the test running instead of calling it early.

Read the full walkthrough in how to A/B test landing pages, or start for free.

Frequently asked questions

How long should a Shopify A/B test run?
Run it for at least two full weeks, so weekday and weekend shoppers both count. Then keep going until each version reaches the sample size you planned. Most stores land on two to four weeks for a clean result.
Can I split test Shopify checkout pages without Shopify Plus?
Only in a limited way. Shopify restricts checkout customization outside Shopify Plus, so there is little to change on the checkout itself. Test the product page, the cart and the offer instead, since they decide who reaches checkout.
What traffic volume is enough for ab testing retail audiences?
Enough for each version to reach its planned sample within a few weeks. A page converting at 3% needs about 14,000 visitors per version to catch a 20% lift. At 1,000 visitors a day, that takes about four weeks.
Does split testing Shopify pages slow down my site speed?
It can, if a tool swaps content after the page loads. That adds a script and can flash the original page first. Replo splits visitors before the page loads, so nothing flickers. With any setup, check page speed on a phone while the test runs.

Start A/B tests

Replo splits your traffic between two versions of a page and tells you which one sells more.