A/B Testing for Online Stores Without Fooling Yourself

Ecommerce A/B testing means showing two versions of a page or feature to different groups of visitors at the same time, then comparing which leads to more orders or revenue. Done properly, it replaces opinions with evidence and protects you from changes that look good but quietly lose sales. Done carelessly, it produces confident conclusions from random noise. The difference comes down to a few disciplines: testing changes that matter, having enough traffic, choosing the right measure before you start, running tests long enough and reading the results honestly, even when they disappoint the person who suggested the idea.

What A/B testing can and cannot tell you

It helps to be clear about what a test really proves. An A/B test answers one narrow question: did version B perform better than version A, for these visitors, during this period, on this measure? It is powerful because both versions run at the same time, so seasonal changes, campaigns and news affect both equally.

It cannot tell you why a version won, whether the result will hold forever, or whether a completely different idea would have done better. Usability research from the Nielsen Norman Group on A/B testing explains why tests work best alongside qualitative research, such as watching people use the store.

There is also a cost to every test: design and development time, a period where some visitors see a weaker version, and the attention of the team. That is another reason to test only changes that could make a real difference.

Testing is also not the only way to improve. Obvious bugs, broken pages and missing information should simply be fixed. Save tests for changes where the outcome is genuinely uncertain.

Choosing what to test

The best tests come from evidence: analytics showing a large drop-off, recordings showing confusion, customer service hearing the same question again and again. Our guide on how to improve your conversion rate explains how to find these problems.

Good candidates include:

• How delivery costs and dates are shown on product pages.
• The layout of the product page on mobile.
• The number of steps or fields in checkout.
• How filters work on category pages.
• The default sort order on large categories.
• Free-delivery thresholds and how they are communicated.

Prioritise pages with the most traffic and the clearest problems, because results arrive faster there and improvements affect more customers. Weak candidates are tiny cosmetic changes, such as a slightly different button shade, which rarely change behaviour enough to measure on a typical store.

Write a hypothesis

Before building a test, write it down: "Because recordings show mobile shoppers scrolling back to find the price, we believe a sticky price and add-to-cart bar will increase mobile orders." A clear hypothesis makes the test purposeful and helps you learn even if it fails. If the sticky bar makes no difference, you have learned that finding the price was not the real problem, and you can look elsewhere.

Traffic: the limit most stores hit

To detect a real difference, a test needs enough visitors and enough orders in each version. The smaller the difference you hope to detect, the more traffic you need. Many small and medium stores simply do not have enough orders to measure small improvements reliably within a sensible time.

Traffic also has to be the right traffic. A test on a product page that few visitors reach will take far longer than one on your busiest category, even if the idea is equally good. Use a sample size calculator before starting to estimate how long a test must run. If the answer is many months, test a bolder change or improve without formal testing.

Choosing the right measure

Decide your primary measure before the test begins. For most store tests, the right measures are close to money:

• Conversion rate: orders per visitor.
• Revenue per visitor: combines conversion and order value.
• Average order value, for tests about bundles, thresholds or upsells.

Clicks, add-to-cart rate and time on page can be useful supporting measures, but they can mislead. A change that increases add-to-cart while reducing completed orders is not an improvement. Also watch side effects such as returns, refunds and support contacts where a change could affect them. A test that raises orders by pushing shoppers into unsuitable products can cost more in returns than it earns.

Running tests properly

1. Split traffic randomly and keep each visitor in the same version throughout.
2. Run for full weeks, ideally at least two, so weekday and weekend behaviour are both included.
3. Do not stop early because one version looks ahead after a few days; early leads often disappear.
4. Avoid overlapping tests on the same page or journey, which make results hard to interpret.
5. Check both versions work on all devices and browsers before starting.
6. Watch for external events, such as major sales or outages, that could distort results.

Check the split as the test runs. If one version receives far more visitors than the other, or if orders from one version are not being tracked, stop and fix the set-up before trusting any result.

Reading results honestly

Agree in advance how you will decide, so the result is not interpreted to suit whoever proposed the idea. When the test ends, look at the result for your primary measure and how confident you can be in it. Testing tools report this as statistical significance or probability; treat borderline results with caution.

• Clear win: roll the change out, then keep monitoring to make sure the effect holds.
• Clear loss: keep the original, and record what you learned.
• No clear difference: the change probably does not matter much; choose based on cost, simplicity or brand.

Look at device and traffic-source breakdowns, but be careful. Slicing results many ways will eventually find a "winning" segment by chance. Treat segment findings as ideas for new tests, not conclusions. If a change seems to help only one group, test that group specifically next time round instead.

Common mistakes that mislead stores

• Stopping tests as soon as they look good.
• Running tests with far too little traffic.
• Measuring clicks instead of orders.
• Changing the test halfway through.
• Testing many tiny variations at once without enough traffic.
• Copying another store's winning test without checking it fits your customers.
• Forgetting to remove old test code, which slows pages.

Most of these come from impatience or pressure to show a win. A culture that values learning, including from tests that fail, produces far more reliable improvements over time.

Ecommerce A/B testing: a worked example

A homeware store notices that mobile visitors add products to the cart far less often than desktop visitors. Session recordings show mobile shoppers scrolling up and down long product pages, apparently looking for the price and delivery information. Customer emails often ask about delivery costs.

The hypothesis

"Because mobile shoppers struggle to find price and delivery information on long product pages, adding a sticky bar with the price, delivery estimate and add-to-cart button will increase mobile orders."

The test

Half of mobile visitors see the current page; half see the version with the sticky bar. The primary measure is mobile orders per visitor, with revenue per visitor and returns as secondary checks. Before starting, the team uses a sample size calculator and decides to run for four full weeks.

The result

After four weeks, the new version shows a clear increase in mobile orders, with no rise in returns. The team rolls it out to all mobile visitors, keeps monitoring for a month and records the result in its test log, along with a note that delivery information placement may be worth testing on desktop too.

The value came not from the bar itself but from the process: evidence, a clear hypothesis, a sensible measure and patience. Our list of mobile UX mistakes is a good source of similar test ideas.

Building a testing habit

One test rarely transforms a store. A steady habit of testing and learning does.

• Keep a backlog of test ideas, each linked to evidence and scored for impact and effort.
• Run one well-designed test at a time on each key page or journey.
• Review results together with marketing, merchandising and development, so everyone learns from them.
• Keep a test log with the hypothesis, dates, result and decision for every test.
• Revisit old winners occasionally, because customers, devices and competitors change.

Reliable tracking is the foundation of all of this. If your analytics do not record orders and revenue accurately, your test results will not be accurate either. Our guide to ecommerce analytics with GA4 explains how to set tracking up properly.

Frequently asked questions

Want testing that leads to real improvements rather than misleading numbers? Talk to our team about conversion research, testing set-up and the store changes that follow.