« »

Monday, July 21, 2014

Boosting conversion rate and ROI with split testing - Mastering the basics

Did you know that you can run a split test showing statistical significance - but see no increase in performance? Why is this the case?

More and more companies are taking a data-driven approach to their marketing, sales, and even their content marketing. However, I find that, while more and more people are measuring and getting strong results, there isn’t a lot of material covering the basics and nuances of testing.

In this article, I’ll cover the basics of split testing in detail, including how to be absolutely sure that your tests are producing real results and how to truly calculate the ROI of your testing.

What is split testing?

At its most basic level, split testing is a systemic way to determine how variables on your website affect conversion and performance. Note that it’s not meant to be a random process, but rather a method - in practice, it’s a combination of art and science.

I’m sure that many of you have run through split tests before, but even with that, let’s walk through a simple A/B test from Ion Interactive:
Conversion rate and ROI split testing

In this test, the main variable being tested is the headline design. For best practice, all other variables are kept the same on the page so that we can isolate the impact of this *one* change.

Splitting traffic & measuring results

Now that we have our testing versions, how do we split traffic between these two sites? How do we determine at a certain level of confidence that one version of the site is performing better than other?
Determining whether your split test was truly significant seems simple, but often is not!

1. Statistics (Chi Square)

The standard approach is to compute a chi square test - specifically, a slightly modified form of a chi square test called an N-1 Chi Square test. (This kind of stats geekery is the reason why most CRO professionals tend to be math geeks at heart). Here’s an example of how it works:
  • The original design had 20,000 unique visitors converting at 5%. This led to 1,000 total actions/mo
  • We run a split test using 50/50 traffic: 10K unique visitors to the control and 10K unique visitors to the test.
  • We determine using 1 month of data that the control is performing at a conversion rate of 6% vs the original performing at a conversion rate of 5%.
Armed with this data, most people will do the logical thing here and simply plug these numbers into an A/B testing calculator, where they’ll get an output showing 99.98% confidence. 99.98% confidence?? Boom! Are we done?

2. Measuring real revenue

Just because a statistics formula shows a high level of confidence does not mean that you will see a true revenue uplift.

While it’s extremely likely that you achieved a lift, remember a few caveats:
  • These tests are based on a chi-squared distribution. I will not get into the heavy math details here, but remember that while this distribution is a “natural phenomenon”, it is to some degree arbitrary.
  • These tests are based only on current traffic and not on future traffic. I can’t emphasize this enough! If you traffic sources change, you may have a different set of “eyeballs” on your page who are looking for different products or who value different elements in design. In this case, you may have lifted conversion for your original audience, but you didn’t necessarily lift conversion for the audience going forward.
IMHO, the absolute best way for you to run split tests to determine real revenue uplift is to do the following:
  1. First, run a split test and get a statistical significance above 95%
  2. Then, push your winning version of the site live to >50% of traffic and double-check that you are still seeing a lift.
Using a “double-check” can slow down CRO, but when a small % increase in conversion rate can drive such a high revenue impact for your site, it’s worth it to be absolutely sure that you are seeing true revenue uplift.

If you do not do this, you’re going to risk running into a few problems that you can’t explain. For example:

Last month my conversion rate was 5%. I ran a split test showing a 10% uplift at a 99% confidence interval. But, now that I’ve put the test live, I’m only seeing a 4.5% conversion rate. What happened?
Can you see clearly what happened now? It’s one of two things - either:
  • Your traffic sources changed, and you are no longer seeing a lift, or
  • Your landing page is still performing better than the original, but the “base conversion rate” changed. This means that there is something else going on that’s lowered conversion - such as bad PR, poor website speed, etc - so while the new site is still performing better than the old, your total conversion rate has still dropped.
In either of these cases, you would be better off doing a “double-check” on your new page to determine absolutely that revenue has been lifted. Otherwise, you’re not going to know what happened.

Calculating ROI

Like anything else, the time that you spend optimizing your site is an investment, as is any development fees or consulting fees that you use while getting a conversion uplift.

So, let’s say that you’ve run a test showing a 10% lift at 95% confidence. You’ve double-checked the lift and you are seeing at least a 10% improvement. Great! Now, what was the real return on all that time?

Let’s say that you saw $10K in additional profit for $2K cost (including cost of time). You’d calculate an ROI of 4x in one-year. But, this # is likely wrong and understates the true ROI impact.
To really calculate ROI, there are a few very important questions that you need to ask yourself:
  • How long is this revenue lift going to last?
  • What is the Lifetime Value of the new customers that I’ve acquired?
  • How much are you spending on marketing?
I have a few tips for each of these:
  1. The uplift should last at least 1 year for a well thought-out experiment: Some companies - particularly startups - move so fast that a conversion increase may have to be retested every month. But for more traditional companies who have been in business for 10+ years selling a similar product, any conversion experiment should last for about 1 year.
  2. Customer lifetime value is more than you think: The vast majority of businesses underestimate the amount that they can spend to acquire a customer. For example, if you get one customer in the door, how does that affect customer referrals? Does having past customer success stories make it easier to acquire new customers?
  3. You’ll reduce your marketing costs: An increase in conversion rate will make your marketing more “efficient”. I.e. if you were to double conversion, you would cut your marketing in half, and this reduction would last for as long as the conversion increase lasted.
Truly calculating ROI in this case can get a bit complex, but what I want you to remember is that your true ROI from split testing and CRO is often much higher than you think.


To accurately run a split test and measure results, you need to take a careful, well-thought-out approach to measurement. What split tests have you run in the past, and how have you measured results?

About the author: Andy is a nerd for split testing and landing page optimization. He currently works as a CRO expert and spends most of his free time keeping up to date with all the trends in online marketing. Follow him on Twitter @upliftroi to get practical tips on conversion rate optimization.

Image: Author owned and licensed