A/B Test the Right Way: 4 Tips You Should Know

Every digital marketing asset, whether it be a native ad, landing page, or email, is made up of dozens of elements. Copy, image, headline, call to action, colors, placement of buttons, video – the list is long. When there are so many variables involved, optimizing your assets is far from simple.

That’s where A/B testing comes in. A/B testing is a common and effective way to optimize your digital marketing assets. It involves testing of two variants – A and B – to find out which is performing better. Then, you can conduct another A/B test, comparing the winning variant with a new variant, to optimize even further. And on it goes.

A/B testing is not a quick fix. It takes time, effort, and diligence to get results. If you’re running A/B tests, or thinking about starting, there are 4 essential things you really should know:

1. Always Use a Testing Tool

Have you ever thought to yourself, “I don’t need an A/B testing tool because I can just run with version A for a week and then replace it with version B the following week?”

While it’s a tempting thought, you should know – that’s not A/B testing. And if you want to do A/B testing the right way, you’ll need to use a proper testing tool.

In A/B testing, both variations must run side by side, and they must be tested in identical conditions (as much as possible). There are a bunch of factors that can impact the test results, like seasonality, day of the week, and your media buying budget, to name a few. That’s just one of the reasons why – and I can’t say it enough! – A/B tests must be conducted in a proper statistical manner. And the only way to do that is with a valid A/B testing tool.

Tools such as Optimizely, VWO, or Google Optimize, for example, will create the environment needed to run a test with a minimum margin of error (as long as you are paying attention and using it properly, of course!). The tool will alert you if your test doesn’t have enough traffic, how significant your result is, and when it’s safe to stop running the test. If you want to get statistically valid results that will make a real difference to your optimization efforts, then there’s no getting around it – use a testing tool.

2. Test Only One Thing at a Time

So you’re looking at your landing page, and you have no idea what to test first. Now’s the time to create a list of elements you want to test and arrange them in order of priority, because you are not going to test everything at once. Why? There are two reasons.

Firstly, testing one thing at a time is the only way to know how it impacts your KPI. If you test a few elements at once, how will you attribute the results to one change or another?

Secondly, testing often causes a burden to performance that can affect the page’s loading time or cause flickering images to appear on the screen. The more changes you implement, the harder it is for the page to load properly, and the more likely your page performance will suffer.

Make sure to keep the entire testing process orderly and logical. Be sure to generate your testing ideas from data, not from guesswork. You can do this by tracking the behavior of users on your assets. This will provide you with information about what’s not working, and what should be optimized. Then, test each element, one at a time, so you can get an accurate and clear picture from your A/B test results.

3. Start with Low Effort and High Impact Tests

So the list of items you want to test is ready – how do you decide where to start?

By ranking all the items according to two things: effort level and potential impact.

Effort level refers to how hard it is to set up the test, and how much work is required to get it running. A test you can create yourself in the A/B testing tool is “low effort”, while a test that requires design or development work, and the input of other teams, would be “high effort”.

Potential impact refers to how much of an effect the test will have on your bottom line. This will depend on your specific KPIs, such as revenue, new subscribers, app installs, and more. If a test has a stronger influence on one of your leading KPIs, it would be considered “high impact”.

Figure out what to test first by tagging the items on your list according to effort level and potential impact. This will help to clarify your priorities for your A/B testing activities.

But wait, how can you know for sure? You can’t (nothing is 100% certain). To some extent, you’ll be relying on working assumptions, based on your experience.

However, following the “effort and impact” model will make it much easier. Low effort and high impact is the low-hanging fruit, or the quick wins, of A/B testing. Start there, and then you can progress towards more complicated tests.

4. Consider Your Goal Carefully

Most testing tools let you choose a few goals for each test. But it is important to focus on the main goal.

While you may be curious to see how different KPIs stand up to the test, you should be very careful about what you define as the most important goal, because this will have the strongest impact on the test results.

For example, say you are testing the copy on a CTA that takes customers to a lead form. You want to track the number of clicks on the CTA, but you should define the main goal as the number of leads received. In other words, you want the CTA copy to get a good number of clicks. But your key goal (or KPI) is getting leads from those clicks. You can only know whether your CTA copy is performing if you accurately define your key goal.

What’s more, to minimize the chances of reaching the wrong conclusion, it is important to define the main goal of the test according to a KPI that is further down your conversion funnel. When your test is more focused, you are more likely to get a solid, accurate result.

Bonus Tip: A Word About AI

Part of the hard slog of A/B testing is creating all the assets to test. If you are testing headlines or email subject lines, you need to continually come up with creative variations. For image tests, you need to produce images with different elements. Even if the variation is fairly minor – for example, a different background color – it still requires you to do the work. While some of the work of A/B testing is creatively oriented, some of it is simply rote or menial tasks. AI can help with both.

Generative AI tools can generate copy variations or different versions of similar images in seconds. This reduces the creative load, which makes the A/B tester’s job much easier. For example, Outbrain’s AI Headline Generator is built into the dashboard, so advertisers can test variations of headlines for their Outbrain campaigns directly within the platform. Apart from the reduced workload, one of the amazing advantages of AI in A/B testing is speed. AI tools make the process of creating, testing, creating, and testing again so much faster and more streamlined. The faster you can get A/B test results, the faster you can iterate and arrive at truly well-performing ads. 

To Sum Up…

A/B testing is a critical part of any digital marketing activity. You need to be analyzing and optimizing – constantly! – to get the highest ROI from your online assets. A/B testing cuts through the noise so you can understand what’s working and what’s not. It might be the smallest thing, like changing the color of your CTA button. Or it might be a larger change, like rewriting the headline on a landing page.

Whatever it is, you’ll only find out via thorough, methodical A/B testing. And if you’re going to do it right, make sure to follow the four key points outlined above. And save time and energy by giving generative AI tools a try. It’s the surest way to make your A/B tests excel.