Tuesday, July 22, 2014

Newsletter: U.S. marijuana's financial future

Financial news: 07/22/2014

CNBC: Economic risk has been priced in to gold price per analyst
CNN: A leading billionaire thinks 3-day, 11 hour work weeks are feasible
GMO: Economic bubble bursting resolution is inevitable via monetary policy
AP: Botox maker Allergen to layoff 13% of its workforce, 1,500 employees
Reuters: A problematic U.K. bank trading mechanism is losing investors
ZH: U.K.'s government is investigating Forex for manipulation
Bloomberg: Russia's foreign policy is bad for business per its' billionaires
NYT: FDI in Africa is forecast to reach $80 Billion in 2014
BI: Rooftop farms are capable of yielding 10% more than traditional farms
BBC: Fast food retailers in China have had their meat supply disrupted

Monday, July 21, 2014

Boosting conversion rate and ROI with split testing - Mastering the basics

Did you know that you can run a split test showing statistical significance - but see no increase in performance? Why is this the case?

More and more companies are taking a data-driven approach to their marketing, sales, and even their content marketing. However, I find that, while more and more people are measuring and getting strong results, there isn’t a lot of material covering the basics and nuances of testing.

In this article, I’ll cover the basics of split testing in detail, including how to be absolutely sure that your tests are producing real results and how to truly calculate the ROI of your testing.

What is split testing?

At its most basic level, split testing is a systemic way to determine how variables on your website affect conversion and performance. Note that it’s not meant to be a random process, but rather a method - in practice, it’s a combination of art and science.

I’m sure that many of you have run through split tests before, but even with that, let’s walk through a simple A/B test from Ion Interactive:
Conversion rate and ROI split testing

In this test, the main variable being tested is the headline design. For best practice, all other variables are kept the same on the page so that we can isolate the impact of this *one* change.

Splitting traffic & measuring results

Now that we have our testing versions, how do we split traffic between these two sites? How do we determine at a certain level of confidence that one version of the site is performing better than other?
Determining whether your split test was truly significant seems simple, but often is not!

1. Statistics (Chi Square)

The standard approach is to compute a chi square test - specifically, a slightly modified form of a chi square test called an N-1 Chi Square test. (This kind of stats geekery is the reason why most CRO professionals tend to be math geeks at heart). Here’s an example of how it works:
  • The original design had 20,000 unique visitors converting at 5%. This led to 1,000 total actions/mo
  • We run a split test using 50/50 traffic: 10K unique visitors to the control and 10K unique visitors to the test.
  • We determine using 1 month of data that the control is performing at a conversion rate of 6% vs the original performing at a conversion rate of 5%.
Armed with this data, most people will do the logical thing here and simply plug these numbers into an A/B testing calculator, where they’ll get an output showing 99.98% confidence. 99.98% confidence?? Boom! Are we done?

2. Measuring real revenue

Just because a statistics formula shows a high level of confidence does not mean that you will see a true revenue uplift.

While it’s extremely likely that you achieved a lift, remember a few caveats:
  • These tests are based on a chi-squared distribution. I will not get into the heavy math details here, but remember that while this distribution is a “natural phenomenon”, it is to some degree arbitrary.
  • These tests are based only on current traffic and not on future traffic. I can’t emphasize this enough! If you traffic sources change, you may have a different set of “eyeballs” on your page who are looking for different products or who value different elements in design. In this case, you may have lifted conversion for your original audience, but you didn’t necessarily lift conversion for the audience going forward.
IMHO, the absolute best way for you to run split tests to determine real revenue uplift is to do the following:
  1. First, run a split test and get a statistical significance above 95%
  2. Then, push your winning version of the site live to >50% of traffic and double-check that you are still seeing a lift.
Using a “double-check” can slow down CRO, but when a small % increase in conversion rate can drive such a high revenue impact for your site, it’s worth it to be absolutely sure that you are seeing true revenue uplift.

If you do not do this, you’re going to risk running into a few problems that you can’t explain. For example:

Last month my conversion rate was 5%. I ran a split test showing a 10% uplift at a 99% confidence interval. But, now that I’ve put the test live, I’m only seeing a 4.5% conversion rate. What happened?
Can you see clearly what happened now? It’s one of two things - either:
  • Your traffic sources changed, and you are no longer seeing a lift, or
  • Your landing page is still performing better than the original, but the “base conversion rate” changed. This means that there is something else going on that’s lowered conversion - such as bad PR, poor website speed, etc - so while the new site is still performing better than the old, your total conversion rate has still dropped.
In either of these cases, you would be better off doing a “double-check” on your new page to determine absolutely that revenue has been lifted. Otherwise, you’re not going to know what happened.

Calculating ROI

Like anything else, the time that you spend optimizing your site is an investment, as is any development fees or consulting fees that you use while getting a conversion uplift.

So, let’s say that you’ve run a test showing a 10% lift at 95% confidence. You’ve double-checked the lift and you are seeing at least a 10% improvement. Great! Now, what was the real return on all that time?

Let’s say that you saw $10K in additional profit for $2K cost (including cost of time). You’d calculate an ROI of 4x in one-year. But, this # is likely wrong and understates the true ROI impact.
To really calculate ROI, there are a few very important questions that you need to ask yourself:
  • How long is this revenue lift going to last?
  • What is the Lifetime Value of the new customers that I’ve acquired?
  • How much are you spending on marketing?
I have a few tips for each of these:
  1. The uplift should last at least 1 year for a well thought-out experiment: Some companies - particularly startups - move so fast that a conversion increase may have to be retested every month. But for more traditional companies who have been in business for 10+ years selling a similar product, any conversion experiment should last for about 1 year.
  2. Customer lifetime value is more than you think: The vast majority of businesses underestimate the amount that they can spend to acquire a customer. For example, if you get one customer in the door, how does that affect customer referrals? Does having past customer success stories make it easier to acquire new customers?
  3. You’ll reduce your marketing costs: An increase in conversion rate will make your marketing more “efficient”. I.e. if you were to double conversion, you would cut your marketing in half, and this reduction would last for as long as the conversion increase lasted.
Truly calculating ROI in this case can get a bit complex, but what I want you to remember is that your true ROI from split testing and CRO is often much higher than you think.


To accurately run a split test and measure results, you need to take a careful, well-thought-out approach to measurement. What split tests have you run in the past, and how have you measured results?

About the author: Andy is a nerd for split testing and landing page optimization. He currently works as a CRO expert and spends most of his free time keeping up to date with all the trends in online marketing. Follow him on Twitter @upliftroi to get practical tips on conversion rate optimization.

Image: Author owned and licensed

Should you buy with cash or lease your car?

By Matthew Hobbs

Your car is one of the biggest purchases you'll make, perhaps even second only to your home, and therefore it's essential to get the right make and model. However, before you start visiting the showrooms and booking test drives, it's essential to think about how you are going to pay for it, and what your budget is going to be. It could be a good time to buy, with car prices rising below inflation, meaning you could get more for your money, so you could look into buying or leasing a new vehicle. Here are a few of the advantages and disadvantages of paying up front or taking your car out on a leasing agreement so that you can decide what's right for you.
Rental car
Leasing agreements

Essentially, when you lease a car it's a form of long term rental, and some contracts have a certain mileage allowance. This means that as well as finding the right car for you, it's also essential that you find the right agreement to suit your lifestyle. Dealers will often have special offers, but it's essential to read the fine print. You should look at:

  • The initial payment or deposit
  • Monthly payments and whether they increase over time
  • Whether you can pay a lump sum at the end to own the car
  • How a high value car will affect your insurance premiums

Doing the maths

The best way to work out whether you'd be better off with a lease is to look at your usage and the value of the kind of car you will be leasing. Work out the cost of the deposit plus the monthly payments for the leasing term, and this will show you how much the car has cost to own in this time, minus fuel, tax, insurance and the usual running costs. To calculate how much the car would cost you in this period if you buy it outright, then you could look at the depreciation during these months, as this will essentially be how much the car has cost you. New and expensive cars can depreciate in value rather quickly, sometimes by as much as 40 percent, so do your research before you choose the make and model, and you can then decide whether leasing comes out in your favour. 
20 pound notes

Buying with cash

If you have the cash to buy a car outright then it can be tempting to empty your savings. If you use your car for a long commute or are generally a high mileage user then buying outright can sometimes make sense, as it means you're not locked into a contract. However, it can sometimes work out more expensive to own a car this way, and if you want to change your car every few years then you may lose out with the dreaded deprecation.

Those who decide to buy in cash will often find that dealers can be a lot more flexible with the price, and therefore they should do plenty of shopping around to see what offers are on. You'll often find deals such as free insurance, cash back, or other specials, but it's important not to get too sucked in by these offers, as they are often a way of distracting you from higher prices or worse deals.

Used cars

If you have the cash to buy outright then a used car might be an option for you. There are many sellers who put cars that are just a couple of years old, and you can haggle to get the best deal for your money. This can help you to get a great deal on a car that hasn't been used much, and this will help you avoid the major depreciation that happens at the start of a car's life.

It's essential to get the best value when you are looking for a new car, and this can mean choosing between buying or leasing. The best path for you will depend on what you want to buy, what your priorities are, and how much you will use your car, so make sure you do price comparisons and take your time to find the right vehicle.

About the author: Matthew Hobbs writes on a freelance basis for Leasing Options car leasing company, one of the most reliable providers of vehicles for rent in the UK. They partner with more than 30 different manufacturers to bring a wide selection to their clients.

Image source: https://farm4.staticflickr.com/3434/3770340233_c2ef6cac78_z.jpg
Image source: https://farm6.staticflickr.com/5014/5474194477_b8e8ccbb60_z.jpg