A/B testing with Unless

Learn how to use Unless to run experiments, test page personalizations and split traffic using the control group.

First of all, you should know that website personalization is different from A/B testing.

Where A/B testing views your visitors as a homogenous group, personalization is about understanding who those visitors are and how to segment them into audiences. In other words, where A/B testing can help you achieve your "local maxima", personalization can help you reach your "global maxima". This has certain implications for testing.

Example: Let's say you create two personalizations. The first is targeted at visitors from the US and changes the CTA on the homepage. The second is targeted at mobile visitors and changes the headline on the homepage. Some (but not all) of your website visitors will see both personalizations - they're in the US and browse the site using a mobile device. The conversion rate goes up - but what caused the uplift? The changed CTA or the new headline?

As the example above illustrates, the nature of personalization (ie the "allowed overlap") makes data interpretation more difficult than in an A/B test.

You can think of personalization as a sequence of optimizations and data-scientists agree that the best way to reliably measure your personalization(s) ROI is by using a so-called holdback set. The holdback audience is like a constant control group. It is always shown the original version of your website, regardless of how many personalizations you're running.

You can make the holdback larger to get data faster, or shrink it if you’re very confident. We recommend to set it to 5-10% so you can always monitor how well your personalization program is doing.

You can still experiment with content, CTAs, and design, just keep in mind that the performance is measured globally. Consequently, the Insights page breaks down results by goals and audiences (not personalizations).

How to set up an experiment:

Keeping possible overlap in mind, here's how to create an experiment in Unless.

1. Create a personalization

Choose a page (we chose the homepage), create one personalization (we named ours Version B) and choose a target audience (we chose "Everyone").

versionb unless-ab-testing

2. Set up a goal

Create a goal (we created the goal "signup") and set the control group to 50%.


3. Result

50% of the traffic you direct to the page will be sent to the personalization and 50% will see the original, unpersonalized version of your page.

Why personalization experiments are so tricky

Generally, testing your personalizations is a good idea. However, running proper experiments requires a lot of traffic and the more personalizations you test, the harder it gets to reach statistical significance. Also, the longer you run a test, the higher the risk of results polluted by external factors. Lastly, personalizations influence each other, so with every additional personalization it gets harder and harder to pinpoint what caused a dip or uplift in your goals.

As a result, we recommend to put an emphasis on continuous monitoring and to use the holdback set. Visitors assigned to the holdback (aka control group) will see the original version of your website. This way you can always check how your personalizations perform in comparison to the unpersonalized experience.

insights-goals-unless On the Insights page, you can analyze how often each goal was reached for the original, unpersonalized experience (aka your control group) vs. personalized sessions (aka personalization participants)