A/B Testing: A Deep Dive Into The Great Divide

When it comes to running a marketing campaign, sometimes even the greatest marketing minds cannot always get it right the first time. No amount of research or experience will always guarantee success. One saving grace, however, is without a doubt A/B testing.

What is A/B testing?

A/B testing is a powerful optimisation tool in the marketer's toolbox, and while it has been around for a while, it is not one to sleep on. As marketers, we are pretty good at EDMs, copywriting, landing pages and killer calls-to-action. After all, it really is our bread and butter.

One thing we can confidently accept defeat on, however, is mind reading. There is no crystal ball in sight, no manifestation sessions and unfortunately no tarot cards within arm reach. This means that making blanket assumptions about what your audience will engage with is a sure-fire way to blow both your budget and your time.

A/B testing in action

As the name suggests, A/B testing is a process where your audience is split into two groups, and each group is served a different variation of the same campaign. This provides an unbiased and accurate insight into what elements work best in achieving the desired outcome — clicks, conversions and all the good stuff.

In a nutshell, it takes the guesswork out of running a campaign and improves your overall return on investment. It is the closest thing we have to a crystal ball.

What can you A/B test?

The answer is pretty much anything and everything. We recommend narrowing your focus to the variables that will yield the greatest impact on your results. Some of our top variations to test routinely include:

1. User experience features such as the placement and ordering of certain features on your website
2. Design features such as the colour of your call-to-action buttons
3. Email features such as the subject line, emojis, images, delivery time and buttons

When A/B testing is done consistently and for long enough — we recommend at least two full weeks, and longer if you haven't reached the sample size you need (more on that below) — you will gain a better grasp of your target audience and stay ahead of ever-changing trends. Not only will you gain insights, you will be armed with data to back up every future decision.

Make sure your test can actually find a winner

Here is the part of A/B testing that rarely makes it into the highlight reel: many tests never had a realistic chance of producing a trustworthy result. Not because the idea was bad, but because there simply were not enough visitors or conversions to tell the difference between a real improvement and random noise.

Two ideas matter here, and they are simpler than they sound:

• Sample size is how many visitors (and, more importantly, how many conversions) each version needs before you can draw a conclusion. Conversions vary naturally from day to day. If you call a test after a handful of sales, you are often just reading that natural variation.
• Statistical power is the likelihood that your test will detect a real difference if one exists. A low-powered test is like trying to hear a quiet conversation in a noisy room — the difference might be there, but you will not reliably pick it up.

The smaller the improvement you are hoping to detect, the more traffic you need. Subtle changes, like a slightly different button shade, need a lot of conversions to measure. Bolder changes — a rewritten headline, a shorter form, a restructured page — produce bigger differences that are easier to detect with less traffic.

Most testing tools include a sample size calculator. Use it before you launch, not after. Decide how long the test needs to run and stick to it, rather than stopping the moment one version edges ahead. And be honest about the limits: lower-traffic sites often cannot run reliable tests on every page. That is not a failure — it just means choosing tests carefully and using other research methods, which we cover below.

Write a real hypothesis

"Let's test a green button" is not a hypothesis. A real hypothesis has three parts:

1. The change: exactly what you will do differently.
2. The expected effect: which metric you expect to move, and in which direction.
3. The reasoning: why you believe it will work, ideally based on evidence such as analytics, session recordings or customer feedback.

For example: "If we move delivery costs and returns information next to the add-to-cart button, more visitors will proceed to checkout, because exit surveys show buyers are leaving to look for delivery details." Written this way, every test teaches you something — even a losing one, because it tells you whether your reasoning about customers was right.

Where to test first

With limited traffic and time, start where a change has the most direct impact on revenue or leads:

• Checkout and cart: visitors here have already shown strong intent. Friction at this stage — unexpected costs, forced account creation, confusing steps — is expensive.
• Lead forms: for service businesses, the enquiry form is the conversion. Small improvements here flow straight through to your pipeline.
• Paid landing pages: you are paying for every visitor, so improving the page improves the return on every dollar of ad spend. These pages also tend to have concentrated, predictable traffic, which makes testing easier.
• Pricing pages: how you present plans, packages and inclusions shapes how buyers judge value, and pricing pages are often a key decision point.

What actually moves conversion rates

Button colours get a lot of attention, but in our experience the changes that make a real difference are usually more fundamental:

• Fewer form fields: every field you ask someone to complete is a reason to give up. Ask only for what you genuinely need at this stage, and collect the rest later.
• Trust signals near the call to action: reviews, guarantees, security reassurance, delivery and returns information, or recognisable client logos — placed right where the visitor is deciding, not buried in the footer.
• Mobile speed: a slow-loading page on a phone loses visitors before they see your offer. Speed improvements help every visitor, every day.
• A clear value proposition: within a few seconds, a visitor should understand what you offer, who it is for, and why it is better than the alternative.
• Specific CTA copy: "Get my free quote" or "Book a call with our team" tells people exactly what happens next. "Submit" and "Learn more" do not.

When you don't have the traffic

If your site does not have the volume for reliable A/B tests, you can still improve conversion rates using research rather than experiments:

• Session recordings: tools such as Hotjar or Microsoft Clarity let you watch anonymised visits and spot where people hesitate, rage-click or abandon.
• Heatmaps: show where people click and how far they scroll, revealing whether key content and calls to action are actually being seen.
• Exit surveys: a single question as someone is about to leave — "What stopped you from enquiring today?" — can surface objections you had not considered.
• User interviews: a handful of conversations with recent customers (and people who chose not to buy) often reveal more than months of analytics.
• Expert reviews: an experienced conversion or UX specialist reviewing your key pages against established best practice can identify obvious friction quickly.

Use these insights to make confident improvements directly, then monitor the results in your analytics. When traffic grows, the same research becomes the source of strong hypotheses for proper A/B tests.

The bottom line

A/B testing is only as good as the test behind it. Start with a clear hypothesis, test where the stakes are highest, and give every test enough traffic and time to produce an answer you can trust. Where you don't have the volume, research and qualitative insight will take you further than a test that can never reach a result.

Testing landing pages for paid campaigns? Read how to audit your Google Ads account, or talk to us about conversion strategy.