When making marketing decisions, do you listen to your gut or follow the data? For the best results, you should employ a mix of both — create content and messaging you feel will resonate (usually from qualitative data) and then A/B test how it performs.
Many marketers rely solely on gut instinct, hunches, or hearsay from other “marketing experts” when making changes to their marketing strategy. If that sounds like you, this is your sign to cut the guesswork out of your marketing performance with data-driven experiments like A/B testing.
A/B testing provides evidence for which marketing strategies work and which ones miss the mark. You can use these tests to dig deep into what makes your ideal audience tick, and discover the subtle changes that’ll get them to take action.
Keep reading to find out what you should A/B test across your various marketing channels, best practices, and the tools you can use to execute and track results.
In this article, I’ll cover the basics of A/B testing, different marketing channels to A/B test, best practices and suggested elements you can test, and recommended A/B testing tools.
What is A/B testing?
A/B testing, sometimes called split testing, is an experimentation process where you alter an aspect of your messaging or marketing channel to create a variant, then serve both versions to visitors in a randomized sequence to learn which one has a greater impact.
Note: A/B tests examine variations (ie. the color of a CTA button), whereas split testing usually deals with larger design elements (ie. leading to the homepage vs a dedicated landing page).
The premise behind A/B testing is comparing the performance of two or more variations to identify how different marketing strategies or activities perform with an audience. It’s designed to improve user experience (UX) and increase conversion rate optimization (CRO).
The typical steps to run an A/B test are:
- Choose a variable you want to test (copy, colors, images, etc.)
- Identify your hypothesis and the KPIs you’ll use to measure your hypothesis
- Create two (or more) variants of the variable you’ll be testing
- Split your audience or sample group into equal segments for each variant
- Deliver your A/B test variants to our audience
- Analyze performance against your identified hypothesis and KPIs to determine which variant performed best
Listening to marketing best practices from industry experts is helpful, but they can’t tell you what the best strategy is for your unique audience. The only way to determine that is to experiment, and A/B testing is one of the easiest, most effective methods a marketer can use.
Tip: Test one element at a time. Changing more than one variable makes it difficult to attribute results to a single alteration, so you can’t tell which variable had the biggest impact.
What to A/B test today
A/B tests are a powerful way to refine your marketing performance.
Let’s say your Google Ads landing page generates 50 leads at a 2% conversion rate. If your marketing team conducted an A/B test on that page, they could identify the changes needed to increase the landing page’s conversion rate from 2% to 5%.
If you get 500,000 visitors to your website every month, a 2% conversion rate earns you 10,000 conversions, whereas a 5% conversion rate earns you 25,000. That additional 3% of visitors converting could mean the difference between healthy growth and declining sales.
Only one in seven A/B tests produces a winning result. However, every iteration of the test brings you one step closer to boosting your conversion rate and, ultimately, your marketing growth.
You might discover you need to hone your hypothesis, extend the duration of the A/B test, or make bolder changes between your variants. Even failed A/B tests are beneficial, as they make your next experiment even smarter.
Now, let’s take a look at some of the marketing channels where you should run A/B tests, and the tools you can use for them.
1) Your website
Increasing online conversions requires more than a higher number of site visits.
All the effort you put into driving visitors to your site will be wasted if you don’t optimize for conversions. Run A/B tests to identify weaknesses to improve your conversion rate and close more sales no matter your visitor count.
What to A/B test on websites
Every website has unique aspects that owners can A/B test to win more sales. However, here are a few places to:
- Headline copy
- Call-to-action (CTA) button color
- CTA copy
- CTA positioning
- CTA size
- Image size
- Pricing information
- Image subject
- Font
- Site navigation menu
- Free trial availability
- Promotional content placement
- Product page layout
- Live chat availability
- Free shipping
- Promotional content phrasing
- Website tone
- Video placement
- Checkout process
- Free trial duration
- Time-sensitive bonuses
- Contact form length
- Font size
- Power words
- Text length
- White space ratio
- Landing page layout
Best practices for A/B testing landing pages
An important rule to follow is to test one variable at a time. Multiple alterations in one test clouds the results so you’re unable to tell which change had the biggest impact on performance.
When running A/B tests on your website, prioritize the pages where your changes will have the highest impact on your conversion rate. You can identify these spots by looking at the pages with the most visitors and lowest conversion rates. If you can’t optimize the page to bring up conversions, consider redirecting it to a high-converting page, or adding CTAs to lead visitors to those pages naturally.
Pay special attention to the pages closest to the end of your customer journey. These are the pages people typically visit before converting (usually landing pages, product pages, or the beginning of your checkout sequence).
Create a list of variables you could test on each of these pages and begin testing them one by one. On your checkout page, for example, you could test the following variables:
- The length of the checkout process
- The payment methods available
- The length of the checkout form
- Checkout page wording
- Trust signals on the checkout page
- CTA button color
- Checkout page layout
Once you’ve selected a variable to investigate, confirm your hypothesis and decide what each variant will look like. Then, put them to the test (literally). Analyze the results and, if you have a clear winner, make the winning changes live on your site.
Repeat the above process for all the pages and elements on your website.
Website A/B testing tools
There are a number of tools you can use to perform and analyze website A/B tests. Some of my favorites are VWO, Crazy Egg, and Google Optimize.
VWO
VWO is the market-leading A/B testing tool that brands around the world use for marketing experimentation and conversion rate optimization.
With VWO, you can run split tests or multivariate tests — without needing to write any code. This codeless approach makes it easy for your marketing teams to implement A/B tests without developer support. You can test almost any element of your site and use VWO’s AI to generate alternative copy suggestions.
Once the test is complete, VWO can monitor performance by reviewing insights across funnels, session recordings, heatmaps, on-page surveys, and form analytics.
For better organization, you can centralize the hypotheses, data, and insights from cross-team experiments within VWO. Your team can then use VWO Plan to collaborate on A/B tests in a streamlined, efficient manner.
Crazy Egg
Crazy Egg provides a multitude of conversion optimization tools, from session recordings (to observe visitors’ actions as they navigate your website) to snapshots that reveal where visitors lok, scroll, and click.
Their A/B testing tool allows users to make changes through a visual editor that’s easy to adjust. This removes the need for a developer to code in changes.
You can set up what conversions should look like and goals to track. The tool’s ease of use, quick iterations, and automatic implementation of winning variants make it a top choice to run A/B tests.
Google Optimize*
Part of Google’s suite of marketing tools, Google Optimize is a free experimentation tool that can easily run A/B tests on your website.
Use the visual editor to alter elements on your website, then A/B test them to see which variants perform best with your visitors. The tool’s capabilities also include multivariate tests, redirect tests, and server-side experiments.
Google Optimize provides clear, action-oriented insights into the results of current or concluded experiments, revealing how each variant performed against your initial hypothesis and chosen KPIs.
The summary header in the Google Optimize report offers a brief overview of your experiment’s success. This summary instantly shows you the status of your experiment, the number of sessions, start and end times, and recommendations for your test.
*Google Optimize will no longer be available after September 2023. An alternative I’ve found is Optimizely, and you can view other tools in my content tools post.
Bonus: Hotjar
While Hotjar may not have built-in A/B testing features, it’s a valuable tool for analyzing the success of your A/B tests. It can display how site users behave in response to different website variations.
You can use JavaScript to mark up specific variants within the Hotjar recordings. Alternatively, you could integrate Hotjar into A/B testing tools such as Optimizely, Omniconvert, or Google Optimize. This would allow you to monitor Hotjar recordings to see how users behave across variants.
Analyze Hotjar’s heatmaps and recording sessions to identify any behavioral patterns based on the variant visited.
Example website A/B test
Netflix’s intuitive interface isn’t the result of chance or gut instinct. Their secret to creating a seamless streaming experience that users love is a rigorous A/B testing program.
Data scientists and engineers at Netflix A/B test every aspect of the streaming platform, including the interface layout, personalized home pages, and images.
The team at Netflix credit A/B testing for their ability to make informed decisions about how to improve the UX of their streaming platform. Some of their changes result in as much as 20% to 30% more views for their titles.
2) Email newsletter
If your email open rates are slowing down and fewer subscribers are clicking on your links, it might be time to shake things up with A/B testing.
Campaign Monitor’s email click-through rate increased 127% after A/B testing their email design. They repeatedly tested the old design against a different variation and found the new design continually came out on top, leading to a substantially higher click-through rate every time.
A/B testing your email campaigns is a great way to breathe new life into your newsletter and increase those all-important open and click-through rates.
What to A/B test in email newsletters
You can test almost any element, from the subject line to the sender name, use of personalization, or text length.
Here are a few other suggestions to A/B test in your email newsletter:
- Subject line
- Send time
- Plain text vs. html
- Text length
- Text-to-image ratio
- CTA buttons
- Email layout
- “From” name
- Preview text
- Personalization
Even the smallest changes, such as whether or not you include emojis in the subject line, could have a big impact.
Best practices for A/B testing email newsletters
When running A/B tests on your email newsletter’s more permanent fixtures, such as what “From” name gets the most opens, test one list at a time. Subscribers to your product updates, for example, may recognize different names and titles than your marketing newsletter subscribers.
Make A/B tests a regular part of your newsletter strategy, not just a one-off activity. An easy rule of thumb is to test two subject lines to achieve the best open rate.
Most email marketing providers offer A/B testing as part of their service. If yours doesn’t, manually split your subscriber list randomly to ensure a fair testing environment.
Be patient and wait a few days before reviewing the results of your A/B test. A five-day period gives your subscribers enough time to check their emails — making sure you don’t miss out on any potential test subjects.
Newsletter A/B testing tools
Most email marketing software offers A/B testing, making it easier for email marketers to optimize their campaigns. With that said, A/B testing capabilities will differ from platform to platform, so do your research to determine which email marketing provider is best for you.
Let’s look at Mailchimp, HubSpot, and Campaign Monitor as three email marketing tools with A/B testing capabilities.
Mailchimp
Mailchimp is one of the most popular email marketing platforms, especially for small businesses and start-ups. With Mailchimp, it’s easy to conduct A/B tests and optimize your email marketing performance.
When running A/B tests through MailChimp, you can test one variable at a time, with up to three variations. Mailchimp’s A/B testing software is basic, offering pre-defined variables and a step-by-step guide. However, this simplistic process can be beneficial for smaller businesses with tighter budgets looking to improve their campaigns.
HubSpot
HubSpot users can also A/B test their marketing emails to obtain the best results. However, the platform’s A/B testing interface only allows users to test two variations at a time (so no multivariate testing yet).
You can choose which variable you want to test when designing your email variations. This freedom of choice makes HubSpot’s A/B testing program both intuitive and efficient.
Once you’re happy with your variants, you can then amend your testing options by deciding what percentage of subscribers will receive each variation, choosing a winning metric, and amending the duration of the test.
Campaign Monitor
Much like HubSpot, Campaign Monitor also allows its users to A/B test two variations of an email campaign at a time.
Unlike HubSpot, Campaign Monitor’s A/B testing tool allows less personalization. Users first select their primary goal, with options either to increase open rate, click-through rate, or visits to a specific URL.
After choosing a goal, users can then select the variable they want to test. Again, this variable is chosen from a predetermined list based on the primary goal. However, it is possible to test other variables outside of this list.
Users then send the A/B test to their chosen audience segments. After the test is complete, they can monitor the results with an easy-to-read graph that shows the variant’s performance against their selected metric.
Campaign Monitor’s step-by-step process is designed for email marketers of any level to conduct experiments with ease.
Example email marketing A/B test
Airbnb are experts in more than vacation rentals — they also excel at A/B testing.
The team at Airbnb have their own internal A/B testing platform that they use to validate their hypotheses and measure the impact of their experiments. The company thoroughly tests every element of its product features and marketing channels, including email campaigns.
Airbnb’s intelligent A/B testing platform has enabled them to make giant strides in the quality of their user experience as they strategically engineer new features and iterate with greater frequency and control.
One test run by Airbnb’s email marketing team analyzed the email design for emails sent to guests versus hosts to improve performance.
Through these A/B tests, Airbnb learned an “improved” email design didn’t always lead to improved email performance for hosts, despite their initial hypothesis. Even A/B test results that disprove your hypothesis can provide valuable guidance for future marketing decisions.
3) SMS messaging
SMS marketing messages are short, so you need to make every word count.
Considering consumers typically have 192 times more unread emails than text messages, SMS marketing can be a powerful way to reach customers. And with A/B testing, you can improve the success of these messages.
What to A/B test in SMS messages
Although SMS messages have a limited word count and design, you can still add variety to your text marketing campaigns.
A/B test your SMS messages by analyzing:
- Send time
- SMS versus MMS format
- Text length
- Formality
- Emoji use
- Video versus photo
- GIF use
- Offer phrasing
- Capitalization
- Slang and abbreviation use
- CTA copy
- Send frequency for SMS drip campaigns
Best practices for A/B testing SMS messages
When A/B testing your SMS messaging, it’s best to start small. For example, analyze the performance of texts containing emojis versus texts without.
Like email marketing campaigns, begin by testing the SMS messages that are likely to have the most immediate impact. Look for any low-effort, high-impact changes you can start right away. You may, for instance, want to focus on A/B testing cart abandonment text messages to improve the success rate of winning back customers.
Consider which changes would have the greatest effect on your business. Think about the buyer journey stage your SMS message is targeting and how you want to influence those people. If you want to encourage dormant customers to shop with you again, you could test a free shipping offer against an exclusive discount code.
Pinpoint your factors with the highest potential, build hypotheses, and put your variables to the test.
SMS A/B testing tools
There are many SMS marketing, A/B testing, and optimization programs available to enhance the effectiveness of your SMS messages.
Klaviyo, Twilio, and Segment (owned by Twilio) are a few popular tools you can use to start off. Note that many email services also offer SMS A/B testing, such as Campaign Monitor, and vice versa.
Klaviyo
Klaviyo is a marketing automation platform where you can easily create SMS marketing campaigns to acquire, retain, and nurture your customers.
Klaviyo prioritizes data-driven marketing, ensuring customer data informs and drives all your SMS or email campaigns. With Klaviyo, you can send personalized and automated text messages that encourage customers to take action.
Klaviyo’s insights platform helps you uncover the metrics and actions that drive success for your business. Using real-time data, you can see how your brand compares to similar businesses within your industry. You can also dive deeper into each of your SMS marketing messages to see which ones work and which need further optimization.
To A/B test SMS messages with Klaviyo, create a new message campaign, click “Create A/B test,” and amend the variable you want to test. You could test the impact of images, different CTA copy, send time, or something else. The power is in your hands with Klaviyo’s A/B testing tool.
Twilio
If you’re looking for a more advanced personalization tool, Twilio could be right for you. It’s an intelligent customer engagement tool that offers communication APIs to enhance SMS, voice, video, and email marketing channels.
Easily integrate Twilio with your chosen messaging APIs to produce seamless conversations with your customers. With Twilio, you can create SMS campaigns for every stage of the customer journey, from targeted offers to cross-channel conversations.
It also scales alongside your business so you can expand from SMS marketing into MMS, WhatsApp, and multi-channel messaging. Use Twilio to craft highly personalized messages that connect with your audience as well.
Tip: Integrate Twilio with Optimizely to run A/B tests that’ll optimize your SMS campaigns.
Segment
By now, you should know how pivotal customer data is to the success of your A/B tests. Segment is a Customer Data Platform (CDP) that lets you seamlessly collect and leverage customer data from all of your digital channels.
Segment gathers customer data and unifies the customer experience across your marketing platforms and channels in one central location. You can use Segment to target customers with real-time messaging across all channels, including your SMS, emails, website, and mobile app.
Let Segment show you how users behave across various other digital experiences to create informed A/B tests that drive omnichannel success.
4) In-app experience
With more than three million apps on the Google Play Store and more than two million on Apple’s App Store, learning how to optimize your app is crucial for gaining a competitive advantage.
A/B testing in-app experiences can significantly boost your conversion rate. App designers can experiment with the user interface in a variety of ways.
What to A/B test in your app
Much like website A/B testing, there are many elements you can test within the in-app experience, including:
- App design
- Headline
- Images
- Pop-ups and notifications
- Color palette
- CTA copy
- In-app navigation
- Copy length
- Animation use
- Videos
The possibilities with in-app A/B testing are almost endless, so be inquisitive and get creative with your experiments.
Best practices for A/B testing apps
While you want your in-app experience A/B test to be bold enough to notice, don’t make it so different that it’s confusing. Keep changes simple and streamlined, and introduce changes one at a time.
Avoid alterations that will interrupt critical app functions. Users should have no trouble accessing the features they downloaded your app for in the first place; if your customers find your in-app messages too intrusive, they might uninstall it out of frustration.
Focus on enacting changes your audience wants to see that’ll help achieve your company’s as well as customers’ goals. Gather input from customer-facing teams so you understand your customers’ needs and pain points.
Mobile apps are highly complex digital experiences, so if you want to drive impact with in-app A/B tests, you need to test with cadence. It’s unlikely you’ll see an instant uptick in sales after one adjustment. Take Booking.com for example — they have 1,000 experiments running at any time.
In-app A/B testing tools
To optimize your mobile app, you’ll need to equip yourself with intuitive tools. Rather than wildly testing elements with no action plan, you can use a collection of personalization, analytics, and testing tools to make data-driven changes to your app.
Three apps I highly recommend including as part of your in-app A/B testing tech stack are Optimizely, Appcues, and Mixpanel.
Optimizely
Optimizely is an A/B testing tool with a host of experimentation features and services to help businesses unlock their full potential.
Optimizely’s Full Stack Experimentation lets you iterate and test every element of your customers’ digital experience. You can integrate Optimizely with other software, such as Twilio, to conduct A/B tests, then analyze the performance through Optimizely.
Optimizely is 75% faster than other experimentation tools, allowing you to see the impact of your A/B tests even quicker.
Appcues
Appcues is an in-app user engagement platform focused on driving tailored experiences for your customers.
With Appcues, you can easily build and iterate in-app experiences without the need for engineering support or complex coding. This easy-to-use platform offers detailed personalization options and allows you to publish in-app user flows almost instantly.
You can then analyze reports to measure performance and user behavior. Use Appcues to build powerful in-app experiences and customer journeys.
Mixpanel
Mixpanel is an app optimization tool that reveals users’ behaviors and journeys so brands can make informed changes to their app interface.
Offering a powerful suite of analytics, Mixpanel helps brands convert, engage, and retain more users. You can see which features are most popular, dive into user analytics, and pinpoint where users drop off. More importantly, Mixpanel conducts mobile app A/B tests.
App developers can take advantage of its A/B testing platform to go behind the scenes in the app and deploy in-app experiments. You can test anything you want with Mixpanel’s A/B testing platform to ensure you build the best possible app for your users.
Example in-app experience A/B test
The senior manager of email and mobile marketing at realtor.com utilized A/B testing to optimize the company’s in-app experience.
Realtor.com knew making uninformed changes to their mobile app could be disruptive or frustrating for their customers. So, they turned to A/B testing to produce informed in-app changes. Through A/B testing, the company was able to build sophisticated in-app campaigns that reach customers the right way. In turn, its mobile app rate experienced a 10x uplift year-on-year.
Wrapping up — Start A/B testing marketing channels today
Now that you’ve learned the power of A/B testing and have an idea of ways to A/B test your various marketing channels — and the best tools for the job — you can start running your own tests.
Whether you test your website, emails, text messages, or mobile app, the A/B testing fundamentals remain the same. Remember to keep things simple, test one variable at a time, choose a measurable hypothesis, and make bold changes. Test and test again to make sure your marketing channels give your users the best possible experience every time.