A/B Test the Right Way: 4 Tips You Should Know

Gal Nissani
Gal Nissani
|

A/Test tips - Outbrain Blog

Every digital marketing asset, whether it be a native ad, landing page or email, is made up of dozens of elements. Copy, image, headline, call to action, colors, placement of buttons, video – the list is long. When there are so many variables involved, optimizing your assets is far from simple.

That’s where A/B testing comes in. A/B testing is a common and effective way to optimize your digital marketing assets. It involves testing of two variants – A and B – to find out which is performing better. Then, you can conduct another A/B test, comparing the winning variant with a new variant, to optimize even further. And on it goes.

A/B testing is not a quick fix – it takes time, effort and diligence to get results. If you’re running A/B tests, or thinking about starting, there are 4 essential things you really should know:

Use a testing tool.

Have you ever thought to yourself, “I don’t need an A/B testing tool because I can just run with version A for a week and then replace it with version B the following week?”

While it’s a tempting thought, you should know – that’s not A/B testing. And if you want to do A/B testing the right way, you’ll need to use a proper testing tool.

In A/B testing, both variations must run side by side, and they must be tested in identical conditions (as much as possible). There are a bunch of factors that can impact the test results, like seasonality, day of the week, and your media buying budget, to name a few. That’s just one of the reasons why – and I can’t say it enough! – A/B tests must be conducted in a proper statistical manner. And the only way to really do that is with a valid A/B testing tool.

Tools such as Optimizely, VWO or Google Optimize, for example, will create the environment needed to run a test with minimum margin of error (as long as you are paying attention and using it properly, of course!). The tool will alert you if your test doesn’t have enough traffic, how significant your result actually is, and when it’s safe to stop running the test. If you want to get statistically valid results that will make a real difference to your optimization efforts, then there’s no getting around it – use a testing tool.

Test only one thing at a time.

So you’re looking at your landing page, and you have no idea what to test first. Now’s the time to create a list of elements you want to test, and arrange them in order of priority, because you are not going to test everything at once. Why? There’s two reasons.

Firstly, testing one thing at a time is the only way to know how it impacts your KPI. If you test a few elements at once, how will you attribute the results to one change or another?

Secondly, testing often causes a burden to performance that can affect the page’s loading time or cause flickering images to appear on the screen. The more changes you implement, the harder it is for the page to load properly, and the more likely your page performance will suffer.

Make sure to keep the entire testing process orderly and logical. Be sure to generate your testing ideas from data, not from guesswork. You can do this by tracking the behaviour of users on your assets. This will provide you with information about what’s not working, and what should be optimized. Then, test each element, one at a time, so you can get an accurate and clear picture from your A/B test results.

Low effort and high impact goes first.

So the list of items you want to test is ready – how do you decide where to start?

By ranking all the items according to two things: effort level and potential impact.

Effort level refers to how hard it is to set up the test, and how much work is required to get it running. A test you can create yourself in the A/B testing tool is “low effort”, while a test that requires design or development work, and the input of other teams, would be “high effort”.

Potential impact refers to how much of an effect the test will have on your bottom line. This will depend on your specific KPIs, such as revenue, new subscribers, app installs, and more. If a test has a stronger influence on one of your leading KPIs, it would be considered “high impact”.

Figure out what to test first by tagging the items on your list according to effort level and potential impact. This will really help to clarify your priorities for your A/B testing activities.

But wait, how can you really know for sure? You can’t (nothing is 100% certain). To some extent, you’ll be relying on working assumptions, based on your experience.

However, following the “effort and impact” model will make it much easier. Low effort and high impact is the low hanging fruit, or the quick wins, of A/B testing. Start there, and then you can progress towards more complicated tests.

Consider your goal carefully.

Most testing tools let you choose a few goals for each test. But you really want to focus on the main goal.

While you may be curious to see how different KPIs stand up to the test, you should be very careful about what you define as the most important goal, because this will have the strongest impact on the test results.

For example, say you are testing the copy on a CTA that takes customers to a lead form. You want to track the number of clicks on the CTA, but you should define the main goal as the number of leads received. In other words, you want the CTA copy to get a good number of clicks. But, in fact, your key goal (or KPI) is getting leads from those clicks. You can only know whether your CTA copy is really performing if you accurately define your key goal.

What’s more, in order to minimize the chances of reaching the wrong conclusion, it is important to define the main goal of the test according to a KPI that is further down your conversion funnel. When your test is more focused, you are more likely to get a solid, accurate result.

To sum up…

A/B testing is a critical part of any digital marketing activity. You need to be analyzing and optimizing – constantly! – to get the highest ROI from your online assets. A/B testing cuts through the noise so you can really understand what’s working and what’s not. It might be the smallest thing, like changing the color of your CTA button. Or it might be a larger change, like rewriting the headline on a landing page.

Whatever it is, you’ll only find out via thorough, methodical A/B testing. And if you’re going to do it right, make sure to follow the four key points outlined above. It’s the surest way to make your A/B tests excel.

 

Amelia's Recommendations

Take your native advertising to the next level.

Learn more
Gal Nissani

Gal Nissani

Gal is the Conversion Optimization Director at Outbrain where her experience in data analysis, SEO and user experience all come together to make her an integral part of the self-serve team. Her past roles at Tomedes and Yehoshua TBWA represent a perfect medley of marketing and management which she uses to support all of her current initiatives.

Add A Comment

* Your email address will not be published. All fields are required.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  • Jake Blumes| April 11, 2019 at 10:10AM

    Excellent summary. We’re constantly testing and constantly improving our approach. We did test different CTA buttons on our landing page which lead to a qualification form and considered the winning CTA button to be the one that generates more clicks. But, as you say above, the real goal is people who then qualify after they click. Setting goals, the right goals, is important!

  • Jake Blumes| April 11, 2019 at 10:10AM

    How do you reconcile testing situations where you find a better image in the header and then you find a better title at the top of the page. Do you try the old image with the new title even though the new image won and the new title won in separate tests?

  • Gal Nissani
    Gal Nissani| April 15, 2019 at 2:02AM

    Thank you, I’m glad this was helpful.
    Exactly! Could be interesting to re-test with a goal that is further down the funnel and see if you get the same winner.

  • Gal Nissani
    Gal Nissani| April 15, 2019 at 2:02AM

    Hey Jake,

    Great question that refers directly to why we should only be testing one thing at a time.
    Sticking to that methodology basically means you won’t be caught in the situation described.
    If we use your example, you test one change (let’s say it’s the image), and if the new variation wins – you implement it and this now becomes your new “control”. You then move forward to test the next change (the title) and so on.

    Primary step to these tests would be to decide wether to go with the image or title first, this is where the impact and effort analysis can be helpful.