How to run an A/B test

In our previous article, we went over the basics of A/B testing: what it is, when you should consider it, and more. But though you now know what it is, you may also be wondering how to run an A/B test. 

We’ll be upfront with you: experience has taught us that the best way to run an A/B test is with dedicated A/B testing software or a trusted and experienced partner. Platforms like Google Analytics and paid plans from Hubspot can help you run A/B tests on assets such as emails and websites. A partner can help you not only handle the technical steps at the beginning (such as setting a measurable goal and splitting your audience), but they can also help you understand the results when the test wraps up. If you’re not looking at technical marketing statistics day in and day out, running a test by yourself could be overwhelming. 

But keeping that in mind, we can still give you a good idea of what to expect! Bringing this knowledge to a meeting with your advertising partner can help things run more smoothly for both of you. So without further ado, here are the basics of how to run an A/B test. 

1. Start with a goal.  

Remember learning about the scientific method in elementary school? A/B testing is kind of like that! You’ve got to have your hypothesis, your variables, and your experiment. So before you get started with making changes to your ads or landing page, it’s helpful to take a step back and figure out exactly what your intention is. Are you hoping to increase conversions? Decrease the bounce rate? Get more clicks? The important thing is to have a specific metric to measure, whatever it may be. Without a concrete goal, you’ll be unable to truly measure your success.

2. Pick only one variable. 

It can be tempting to come up with all sorts of ideas of variables to change at once. But if you make your creatives too different, you won’t know which change caused the increase in performance. For example: let’s say you’re offering your audience a free houseplant starter kit. if you change the CTA on your ads from “Learn More” to “Get Your Free Kit”, and change the photo used in the ad, and make the color scheme green instead of blue, you might assume the color change made the difference and set about changing your entire website…when it was actually the more active CTA that got more clicks.

Here’s a small sampling of possible variables you could test: 

• Call to action (CTA) 

• Email subject line 

• Landing page layout 

• Creative messaging 

The possibilities are endless – and remember, you can always run multiple A/B tests! In fact, running multiple A/B tests over time will give you a very clear idea of what works for your audience. Just make sure to focus on one variable at a time for the clearest results. 

3. Create your challenger.

Once you’ve identified your goal and your variable, you’ll need two groups: the control or “A” group (which consists of whatever you’re currently using, such as a set of creatives or a landing page) and the challenger or “B” group (which consists of the group with the variable changed). For the “A” group, congratulations, your work is done! For the “B” group, you’ll need to make the new version with the change, whether that’s an ad, a site, or an email. If you’re testing whether your ad performs better when linked to your homepage vs. a dedicated landing page, now is the time to build the landing page.

4. Split your audience.

For best results, your audience should be split equally and randomly. Splitting them equally will ensure that you’ll be able to compare the results of the two groups in a 1:1 fashion. No need to adjust the math for bigger or smaller groups – always a good thing! This is the stage where the steps become more technical and are better handled by someone with experience. Dividing your audience is a task best handled by a trusted partner or by your A/B testing software of choice.

5. Run the test…

Your assets are ready, your audience is split – it’s off to the races! Make sure that your “A” and “B” groups are running at the same time to ensure you’re not introducing another variable. (The one exception here would be if you are testing the timing itself, such as testing what’s the best time to send an email.)

6. …and keep it running long enough to get meaningful results.

Remember to be patient – don’t be too hasty to draw conclusions! There’s no magic formula to determine how long your test should run. But running a campaign for too short of a time can leave you with a skewed idea of your actual results. Keep in mind: if you don’t get much traffic to begin with, an A/B test will take longer to provide you with statistically significant results. If you have massive reach, you’ll be able to get meaningful results sooner. Since most people fall somewhere in the middle, it’s a bit of a balancing act. This is another step where an expert partner can provide you with guidance. Hubspot, for example, has a free A/B testing calculator that can help you determine the statistical significance of your results. Otherwise, your advertising partner can be a big help here. 

7. Analyze your results and plan for the future.

Now that your test is over, it’s time to see what you can learn from it. Sometimes, there will be a clear winner – that makes your job easy! At that point, you can simply disable the losing group and focus your time and money on the winner. Other times, you may need some additional help. If your results are less of a slam dunk, you may need to consider other changes you could make that would move the needle for you. Maybe it wasn’t your CTA that was the problem – maybe it was the photo you used in your ad. You never know until you try! Regardless of what your results turn out to be, you can take that information with you as you plan future marketing efforts.  

And that’s it! Now you know how to run an A/B test. It’s a lot of information, but at its core, an A/B test is really just a simple experiment. If the learning curve or upfront cost have scared you off in the past, we hope this article has helped put your mind at ease. Running A/B tests can help you improve your marketing efforts and reach your audience in the most effective way – and that’s priceless.