We take a look at why A/B testing hasn't lived up to its promises for many. Here is an alternative to A/B testing: personalization.
A/B testing, the darling of the conversion optimization world, has lost its mojo of late. Optimizely’s recent decision to drop its free plan, which provided around 70,000 websites with A/B testing capabilities, is one of the many moves pointing to the industry’s shift from A/B testing to other forms of conversion optimization. Let’s unpack why A/B testing is proving to be ineffective for most businesses and why website personalization is a more viable alternative.
A 2014 survey among VWO customers showed that only 1 in every 7 A/B tests is a statistically significant winning test. This means that at most 14% of A/B split tests result in conversion improvements.
Why is that? For an A/B test to be statistically significant, you need sample sizes that are large enough. Consider the following example.
Let’s do some math and run a hypothetical test that will properly detect any difference at least 80% of the time (also referred to as “statistical power”), assuming that a significant difference between the tested versions actually exists. To be sure, the test will show only 5% of false positives. You could say this is the bare minimum of statistical significance (95%).
With these boundary conditions, if you assumed an increase of your baseline conversion from 4% to 6%, which is a pretty high uptake of 50%, then you would need 1,600+ people to visit each variation. And it gets worse. In real life, improvements are typically smaller than 50%, increasing the required number of visitors dramatically. For example, let’s say you believe you could increase your conversion rate from 4% to 4.4%, which is a relative uptake of 10% and far more realistic, then you would need a staggering 38,000 visitors on each variation.
In other words, you need a heck of a lot of visitors for A/B testing to be statistically significant. But even if tests reach statistical significance, there’s a lot of other things that can go wrong.
A lot of people make blunders when running A/B tests. Peep Laja at CXL, has written extensively about this. Besides calling the results too early, before tests reach statistical significance, common mistakes people make include:
It’s difficult to avoid these mistakes without working with a specialized agency, breaking your wallet on expensive software, or being an experienced analytics whizz.
But perhaps the most important drawback of A/B split testing is that you are operating under the assumption that there can only be ONE winner.
In A/B testing we tend to forget that crowning a winner also means there are losers. A winning variation may have the best result on average, but it may not be the best for all your audience segments. Some people preferred the losing variation because it was more relevant to them. So why would you force the winning variation on everyone? Why miss out on conversions?
Suppose you sell flowers online and Mother’s Day is coming up. You have identified three audience segments, based on interest and purchase history. The first group enjoys classical bouquets, the second loves modern flower arrangements, while the latter prefers potted plants over bouquets. With personalization, you can create a tailored experience for each of those groups — without having to set up three campaigns with three separate landing pages.
One landing page that dynamically changes into three experiences.
In an unsegmented A/B test neither option would stand a chance against a generic, one-size-fits all landing page (“Shop for Mother’s Day!”), yet the combination of these three variations outperforms the original one.
By now you are probably asking yourself how personalization applies to your business. After all it’s not always as easy to identify audience segments as in the example above. To start, you should figure out which types of website personalization are relevant for your business. Generally, they fall into two categories:
The type of personalization you choose will vary depending on your budget and needs. Let’s look at some use cases to make this less abstract.
Personalizing for new vs. returning visitors can result in significant gains from a UX and conversion perspective. Typically, new visitors need more guidance. In B2B this could mean showing an explainer video to first-time visitors and focusing on getting their contact details. Conversely, returning visitors need to be encouraged to schedule a demo with sales, while existing customers should be re-engaged by informing them about the latest feature updates. In eCommerce, personalization typically means a more generic site for first time visitors, and behavior-based content for returning shoppers.
Another great use case is personalizing based on location. Does your business operate globally or locally? Let your visitors know quickly —nothing hurts a conversion rate more than uncertainty.
In B2B this could mean showing social proof from local companies while in B2C you could opt for localized offers.
Are you in the travel business? Personalizing your offer based on the visitor’s current location increases CTR by 35–40%.
Getting started with personalization is simpler than you may think, especially if you have an experienced partner by your side. At Unless, we like to do an onboarding session with new customers, as we know that figuring out “what to personalize?” is the hardest part. The technical implementation is comparably simple — install a script and you are good to go.
To summarize, instead of optimizing for the majority of your site visitors, and losing out on potential conversion, tailor your content for key audiences, increasing the overall effectiveness of your marketing.
Let me know what your thoughts are and check out this website for an overview of currently available personalization solutions.