The ROI of AB Testing: When is AB testing worth it?

Posted By Devesh Khanal | 2 Comments


I have a simple 2-step criteria for determining if AB testing has a high enough ROI to be seriously considered for your marketing team:

  1. Criteria #1: Your business makes $2,000,000 or more in revenue
  2. Criteria #2: You get 100,000 monthly unique visitors or more to your site

Yes, like any numerical “cutoff”, these numbers are somewhat arbitrary (e.g. In the U.S. at 15.5 years old, you are too young to drive, but at 16.1 years, you’re not).

But, when you zoom out, cutoffs have reasoning behind them: You wouldn’t trust an 8 year old to drive, and waiting until people are 30 to drive is also ridiculous.

Similarly, instead of debating the numbers themselves, let’s discuss the reasoning so you can adjust the above numbers as needed for your business. (Fortunately, unlike a driver’s license, our 2 step criteria above aren’t hard rules).

Finally, I’ll also discuss a 3rd corollary rule that falls out of the reasoning behind the first two: If you fulfill the first two criteria and begin investing in AB testing but aren’t getting routine lifts in orders or sales of 5% – 10% or more, the ROI on your testing investment may also not be worth it.

Why a Revenue a Cutoff? Because AB testing isn’t free.

AB testing isn’t free. Even if you include software costs, the majority of AB testing costs are for the people needed to run the operation. This can come in two forms:

  1. An outside agency, that charges you a monthly rate. From my un-scientific survey of other CRO agencies, they typically charge between $2,000/m – $15,000/m, with most experienced ones being typically towards the middle/high end of that range: $5,000 – $10,000/month.
  2. Using internal employees, which also burn cash (almost always more than an agency).

I’ll soon have an entire article that dives into these costs and discusses how to choose between using in-house resources vs. outside. But for now, as an example, say your monthly AB testing costs are $5,000.

Deciding if a dedicated CRO program is “worth it” at $5,000/month is a matter of estimating:

  1. How much of a revenue lift you’re likely to achieve
  2. How long you’ll have to spend to achieve it

Before you get worked up about this, let me make this clear: it’s impossible to know the answers to these questions a priori.

But luckily, you’re not the first company to do AB testing, so we can look at what’s typical, conservative, aggressive, etc., based on past experience.

Because this is all speculation, however, I’m going to just boil things down to this general rule: assume you can get (1) a 10% increase in revenue in (2) 6 months of testing.

So, that means you’d spend $30,000 ($5,000/m for 6 months) and, if you were making $2 million in revenue before (bare minimum of the criteria above), you’ve increased annual revenues by $200,000/year.

That’s a 567% ROI from the first year of revenues alone. Pretty good.

At this point, let me re-iterate a point I made above about arbitrariness in different words: You may not get a 10% lift in 6 months, or, you may get way more than a 10% lift in revenue in 6 months, or you may get the lift in 2 months.

I can’t say.

You can’t say.

No one can predict this.

But, I know that 10% revenue lift in 6 months is reasonable and achievable — our agency has achieved this multiple times for multiple businesses.

If you want to discuss whether this is achievable for your business and possibly get a few initial ideas to test from us, click here.

Now, similar to the driver’s license example, instead of arguing about whether you you should get a 10% or 25% or 7.72% lift in this time period, let’s zoom out and look at some extremes.

If you told me “For a 6 month AB testing campaign, we’ll need to see a 150% increase in sitewide revenue for it to be worth it.” I’d respond with “Maybe you should focus on something else.”

I’m not saying that’s impossible.

Anything is possible.

It’s possible you could do some user research for a month or two, discover some immense barrier to purchasing that was due to on-site elements, UI/UX, or product related issues, fix them, and see an increase in revenue of 150%.

That is possible.

But it’s just not common, and I’d go as far as to say it’s not even reasonable to expect 150% increase in revenue in a few months.

But 10%? That’s reasonable.

Examining Different Revenue Ranges

Revenue of $500,000 – $1,000,000

Let’s look at lower revenue numbers to see why I put our arbitrary criteria at $2 million.

At $500,000, 10% is $50,000 a year. That’s awfully close to the $30,000 you’d spend doing this for 6 months. Sure maybe you’d see that 10% lift in 3 months, but would you then stop your testing spend just to make sure your ROI was good? No, you’d keep going.

Or, you may only see 5% lift, which is only $25,000.

Either way, it’s not likely that at $500,000 sitewide revenue you’re going to get an outstanding, no-brainer ROI on AB testing.

After months of testing, you may only see an increase in revenue of between $50,000 – $150,000, and spend about $60,000 a year in AB testing costs. So although the ROI could be positive, it’s not a no brainer.

More importantly, for businesses in this revenue range, there are often bigger wins.

In my experience, these bigger wins are usually SEO or paid media optimization. Those two drivers of traffic can move the needle significantly.

I’ve seen in multiple client’s analytics, traffic double from one year to the next. If that fruit is still hanging for your business, pick it first. Don’t worry about trying to eek out 10% or even 35% lifts via CRO.

(Yes, it’d be ideal to do both, but resources are finite.)

Or, your business may be hitting the $500,000 or $1,000,000 revenue mark with some initial paid ad spend that is by no means saturated. If that’s the case, can you double paid ad spend and still maintain sufficient profitability?

The answer may be yes, the answer may be no, but that question should be asked and explored in great detail before starting up a CRO program from scratch.

Revenue Less than $500,000

Startups that are just starting to make money often ask us about AB testing.

If your company is making less than $500,000 in revenue, regardless of how much traffic you have, it’s hard to make the math work for CRO. If you make say $250,000/year, you could easily spend 6 months and $30,000 to get a 10% lift and you wouldn’t be making your money back.

Aside: If you’re thinking, “I’m the founder and I’m barely paying myself,” stop. Your time is the most valuable, and almost always at this stage your focus should be on getting better product market fit.

More importantly, even if you did make your money back (or had a positive ROI) from CRO, it’s not likely to produce a step change in growth. The increase will be incremental, and your business is likely at a stage where you’re looking to find significant growth — so focusing there makes more sense. (Typically that opportunity is better product/market fit or traffic generation.)

Revenues of More than $10,000,000

Now, let’s look at 8 figure businesses and above with the same criteria.

A 10% lift, achieved in 3 – 6 months (reasonable), would yield over a $1,000,000 in annual revenue.

You’re not likely going to spend anywhere near $1,000,000 for 3 – 6 months of testing. Even if you paid $10,000 a month to an agency for 6 months, your ROI is significant: 1567% on a $60,000 investment.

Alternatively, even if you increased revenues by just 5%, that’s $500,000 of extra annual revenue — a 733% ROI.

This math, combined with our observation that 8 figure businesses have more often than not spent years optimizing traffic and paid advertising (and thus either don’t have super low hanging fruit or already have a healthy operation focused on cranking out consistent wins on those two fronts), means that starting up a CRO program for an 8-figure+ business is a no brainer.

Note: Even understanding “what is reasonable?” is easier for an established company. They have a traffic history and a paid media history that they can look to. For example: Has your SEO traffic grown at 20% – 30% for the last 3 years? Great, you can probably expect the same this year. 

The word “starting” in the previous sentence is particularly important because like any marketing initiative, the easiest wins are there for the taking at the beginning, so a site that has never before been formally “optimized” likely has some easy wins that can be realized by CRO (user research + hypothesis generation + ab testing) in the first 3 – 6 months.

Corollary Rule #3: You also need to be seeing consistent conversion lifts to get a good ROI on testing

So we see from the above analysis that a certain revenue range is required for reasonable lifts in conversion rate to yield increases in revenue that make testing “worth it”. But that means the inverse is also true: you need to achieve reasonable lifts in conversion rate for testing to be worth it! 

If you check the boxes on the first two criteria:

  1. We have more than $2MM in revenue
  2. We have more than 100,000 monthly uniques

And you start investing in AB testing, great. That means the potential of seeing a great ROI on testing are there. But it doesn’t mean you will achieve that potential.

Continuing with the numbers in my examples above, you’d need to achieve a 10% lift in orders or sales on a consistent basis to realize your ROI potential.

If you are testing with a typical ad hoc or “conference room” approach (everyone sits around a conference room and throws up their favorite pet ideas to test), then it’s quite likely you won’t achieve that result. Feel free to reach out to us at the link below if you want to discuss was to solve this problem in your organization.

Final thoughts

Again, before you start drafting your comment arguing that my numbers are arbitrary, I want to emphasize (for the 15th time) that I’m fully aware they are arbitrary. Don’t use them to the exact digit. Adjust them for your business, and use the guiding principle, which is:

Before investing in CRO, first compare reasonably achievable revenue increases that CRO could produce versus the equivalent increases you could possibly get from other channels or initiatives (e.g. Improving the product, increasing traffic via SEO, paid ads, content marketing, etc.)

Have a 7 or 8-figure business and want to a evaluation of conversion optimization opportunities for your business? Contact us.

Want updates when articles like this come out? Join our email list (at the top of this page).

2 Comments

  1. Gene
    October 19, 2016

    Excellent breakdown! The only thing I’d like to point out is that not meeting criteria 1 or 2 is NOT a good reason to ignore the basics of CRO. Obviously this won’t fall under an agency specializing in generating ROI for businesses meeting your criteria, but a solid one time run through a website is a worthy investment for businesses of any size. Side note: I’m honestly shocked at the number of huge companies I’ve come across who operate without CRO and A/B testing while allowing senior managers to make design decisions based solely on intuition.

    Reply
    • DeveshKhanal
      October 19, 2016

      Great point re: CRO principles applied to smaller sites. We have this post to help on that: http://staging.growthrock.co/conversion-optimization/

      Yes absolutely those site owners should keep in mind best practices, talk talk talk to users and customers, do the user research (polls, session recordings, etc.) make changes and watch data carefully, etc.

      Reply

Leave a Reply to Gene

Cancel Reply