Learn how to use Unless to run experiments, test page personalizations and split traffic using the control group.
First of all, you should know that website personalization is different from A/B testing.
Where A/B testing views your visitors as a homogenous group, personalization is about understanding who those visitors are and how to segment them into audiences. In other words, where A/B testing can help you achieve your "local maxima", personalization can help you reach your "global maxima". This has certain implications for testing.
Example: Let's say you create two personalizations. The first is targeted at visitors from the US and changes the CTA on the homepage. The second is targeted at mobile visitors and changes the headline on the homepage. Some (but not all) of your website visitors will see both personalizations - they're in the US and browse the site using a mobile device. The conversion rate goes up - but what caused the uplift? The changed CTA or the new headline?
As the example above illustrates, the nature of personalization (ie the "allowed overlap") makes data interpretation more difficult than in an A/B test. Nonetheless, many of our customers experiment with content, CTAs, and design. To measure performance, they analyze the Insights page data.
Keeping possible overlap in mind, here's how to create an experiment in Unless.
Choose a page (we chose the homepage), create one personalization (we named ours Version B) and choose a target audience (we chose "Everyone").
Create a goal (we created the goal "signup") and set the control group to 50%.
50% of the traffic you direct to the page will be sent to the personalization and 50% will see the original, unpersonalized version of your page.
Generally, testing your personalizations is a good idea. However, running proper experiments requires a lot of traffic and the more personalizations you test, the harder it gets to reach statistical significance. Also, the longer you run a test, the higher the risk of results polluted by external factors. Lastly, personalizations influence each other, so with every additional personalization it gets harder and harder to pinpoint what caused a dip or uplift in your goals.
As a result, we recommend to put an emphasis on continuous monitoring and to use the control group. Visitors assigned to the control group will see the original version of your website. This way you can always check how your personalizations perform in comparison to the unpersonalized experience.