Disappointed by A/B testing? Time to check out website personalization

A/B testing, the darling of the conversion optimization world, has lost its mojo of late. Optimizely’s recent decision to drop its free plan, which provided around 70,000 websites with A/B testing capabilities, is one of the many moves pointing to the industry’s shift from A/B testing to other forms of conversion optimization. Let’s unpack why A/B testing is proving to be ineffective for most businesses and why website personalization is a more viable alternative.

Only 1 in every 7 A/B tests is a statistically significant winning test

A 2014 survey among VWO customers showed that only 1 in every 7 A/B tests is a statistically significant winning test. This means that at most 14% of A/B split tests result in conversion improvements.

Why is that? For an A/B test to be statistically significant, you need sample sizes that are large enough. Consider the following example.

Let’s do some math and run a hypothetical test that will properly detect any difference at least 80% of the time (also referred to as “statistical power”), assuming that a significant difference between the tested versions actually exists. To be sure, the test will show only 5% of false positives. You could say this is the bare minimum of statistical significance (95%).

With these boundary conditions, if you assumed an increase of your baseline conversion from 4% to 6%, which is a pretty high uptake of 50%, then you would need 1,600+ people to visit each variation. And it gets worse. In real life, improvements are typically smaller than 50%, increasing the required number of visitors dramatically. For example, let’s say you believe you could increase your conversion rate from 4% to 4.4%, which is a relative uptake of 10% and far more realistic, then you would need a staggering 38,000 visitors on each variation.

In other words, you need a heck of a lot of visitors for A/B testing to be statistically significant. But even if tests reach statistical significance, there’s a lot of other things that can go wrong.

A/B testing mistakes are far too common

A lot of people make blunders when running A/B tests. Peep Laja at CXL, has written extensively about this. Besides calling the results too early, before tests reach statistical significance, common mistakes people make include:

• Not ruling out seasonality — i.e. assuming that conversion rates are the same on every day or month. Chances are you run your A/B tests during the week, but what about weekends? The rules of engagement for customers may be different then. It’s important to have a thorough understanding of your conversion rates before you run a test, and it’s equally important to run it for full weeks.
• Not testing based on a hypothesis. As Laja writes, “testing random ideas” is a waste of time. Before running a test, you should do your homework, identify a problem and come up with a hypothesis that could solve it.
• Running multiple tests at the same time — Let’s be honest, most marketers are guilty of running multiple A/B tests at the same time, often with overlapping traffic. The truth is, learning from A/B tests takes time, so it’s important to put quality over quantity.
• Not backing up test results with additional research — this is a mistake almost everybody makes. A/B tests are not the holy grail. Often you have to back them up with usability testing, surveys or heat mapping software to be sure they are reliable.

It’s difficult to avoid these mistakes without working with a specialized agency, breaking your wallet on expensive software, or being an experienced analytics whizz.

But perhaps the most important drawback of A/B split testing is that you are operating under the assumption that there can only be ONE winner.

How A/B testing makes you miss out

In A/B testing we tend to forget that crowning a winner also means there are losers. A winning variation may have the best result on average, but it may not be the best for all your audience segments. Some people preferred the losing variation because it was more relevant to them. So why would you force the winning variation on everyone? Why miss out on conversions?

Suppose you sell flowers online and Mother’s Day is coming up. You have identified three audience segments, based on interest and purchase history. The first group enjoys classical bouquets, the second loves modern flower arrangements, while the latter prefers potted plants over bouquets. With personalization, you can create a tailored experience for each of those groups — without having to set up three campaigns with three separate landing pages.

One landing page that dynamically changes into three experiences.

In an unsegmented A/B test neither option would stand a chance against a generic, one-size-fits all landing page (“Shop for Mother’s Day!”), yet the combination of these three variations outperforms the original one.

Website personalization — a promising alternative to A/B testing

By now you are probably asking yourself how personalization applies to your business. After all it’s not always as easy to identify audience segments as in the example above. To start, you should figure out which types of website personalization are relevant for your business. Generally, they fall into two categories:

• Rules-based personalization is the practice of manually setting certain rules to target different groups of your audience with different content. These rules encompass things like location, weather, time, device type, referral source, whether the visitor is a new or recurring user, etc.
• Automated personalization is personalization through machine learning algorithms, where each variation is automatically crafted for the individual visitor of the website. Decisions are made based on the existing metadata, as well as demographics, and past behavior. This type of personalization sounds amazing but is hard to pull off if you’re not blessed with tons of data — E-commerce sites typically personalize this way.

The type of personalization you choose will vary depending on your budget and needs. Let’s look at some use cases to make this less abstract.

Use case: Optimizing conversions for new vs returning visitors

Personalizing for new vs. returning visitors can result in significant gains from a UX and conversion perspective. Typically, new visitors need more guidance. In B2B this could mean showing an explainer video to first-time visitors and focusing on getting their contact details. Conversely, returning visitors need to be encouraged to schedule a demo with sales, while existing customers should be re-engaged by informing them about the latest feature updates. In eCommerce, personalization typically means a more generic site for first time visitors, and behavior-based content for returning shoppers.

Use case: Optimizing conversions based on location

Another great use case is personalizing based on location. Does your business operate globally or locally? Let your visitors know quickly —nothing hurts a conversion rate more than uncertainty.

In B2B this could mean showing social proof from local companies while in B2C you could opt for localized offers.

Are you in the travel business? Personalizing your offer based on the visitor’s current location increases CTR by 35–40%.

Getting started with personalization is simpler than you may think, especially if you have an experienced partner by your side. At Unless, we like to do an onboarding session with new customers, as we know that figuring out “what to personalize?” is the hardest part. The technical implementation is comparably simple — install a script and you are good to go.

To summarize, instead of optimizing for the majority of your site visitors, and losing out on potential conversion, tailor your content for key audiences, increasing the overall effectiveness of your marketing.

Let me know what your thoughts are and check out this website for an overview of currently available personalization solutions.