I’m often amazed at how naive companies can be about their online testing efforts. The bigger the company, it seems, the more disconnect there can be between the talented optimization team and the senior management. And that, as you might guess, often ends up undermining the revenue and efficiency improvements that come from continuous testing.
This is part 1 of a multi-part column. In future columns, I’ll discuss why you want to celebrate every time you “lose” when testing — which is really what I wanted to write about. But I realized it’s important to lay the groundwork for why continuous improvement through testing is absolutely essential to your company’s online efforts. So that’s where I’ll start.
Most of the arguments I’ve seen “for” testing are made with the marketing team in mind. While this is the correct audience to impact ongoing efforts, the marketers can often be too fuzzy when it comes to their own analytics and measurement. Yet, the analytics team is often overly focused on tools and pure metrics — and, as I discussed in my January 2012 article — focused on too many metrics. This column will use an example, with numbers, to review how testing equals business smarts.
Over the course of this series of columns, we’re going to develop a spreadsheet that you can use to model your own company’s efforts towards testing, each month adding more and more features to the underlying model.
This month, let’s start with the basics. By far, the most common frustration I’ve heard expressed by optimization teams is the high failure rate of their tests. These are the folks in your organization who should know best that “Losing More Often” — a topic we’ll cover in a future part to this series — is actually the recipe for long-term success in continuous improvement.
Introducing Cato’s Kitchen
Let’s start with a hypothetical situation. Imagine your company, Cato’s Kitchen, produces specialty customized cat food — sort of like private-labeling your own bottle of wine. Cato’s is in a tough market dominated by the big guns of the pet food industry, but the concept is certainly a unique approach. And they’re small and nimble, so they can test, change, adapt, and repeat fairly often and at low cost. So their mindset is perfect for improving their online efforts through testing.
Now some numbers: Cato’s team has the bandwidth to do 10 tests a month. And each time they test, the test itself has fixed costs of $1,000 per test, mostly from the simpler tweaks to their website that often accompanies the early phases of testing: low-hanging fruit such as conversion improvement, design tweaks, better calls to actions. Every testing team must begin this way; if your team jumps first to long-term structural change before you handle the simple stuff, find another team.
How about success rates? Cato’s team wins only one out of every ten tests, a win rate of 10%. When they win, it’s worth about $25,000 to the company.
Failure rates? Cato’s team loses about six times out of ten, and when they do it costs about $10,000. And Cato scratches (pardon the pun) about three times out of ten.
Our First Model
Here’s what our first model looks like so far:
Some astute readers will be asking: “Why $1,000 per test as fixed costs?” “Why is the win rate only 10%?” “Why is the average winner 2.5 times as large as the average loser?” Good questions, all!
The Underlying Assumptions
I’m using approximate numbers based on experience. Because after doing this for umpteen years and with eleventy hundred clients, these are the sorts of ratios I’ve seen to be quite common among the most successful testing teams. Their costs for implementing a given change tend to average out to a fairly static fixed amount, typically no worse than 10% of an average losing test, often much lower.
Successful teams’ win rate, while a seemingly low 1:10 on casual inspection, comes from being unafraid of being “wrong” and therefore testing more often (and again, in a future column I’ll get to why you want to test more often and to happily welcome losing tests).
And, finally, successful testing teams typically have average winners that run 2-3 times the size of their average losers. One of the upsides to the “learning what to test” lesson gleaned from continuous testing is that your progressive improvement translates into more efficient winners. This can be reflected in positive improvements to any or all of the key metrics surrounding testing wins: a higher average winner over time, a smaller standard deviation among winners, and outright win rate percentage. All else being equal, by far I would prefer the first since it’s the least subject to post-test second guessing. If you start posting a series of progressively profitable wins, it’ll be hard to argue your team isn’t bringing in true value. (Or so you’d think — I’ll recount a short anecdotal story to the contrary later.)
So I’m pushing our early first draft of a model toward characteristics of teams who win. To be honest, I’m not nearly as concerned about teams that lose; we’re trying to mimic success, not failure. You will have failures along the way, so let’s put up a goal of what success would look like and shoot for that.
The First Month
Now with these numbers in place, we can create a simple spreadsheet to see how we do after a month of testing. (You can download this spreadsheet from the link to my site at the bottom of this column). Here are Cato’s results in the first month:
Let’s step through this to make sure we’re all on the same page. We’re doing 10 tests per month. Each of those ten tests has fixed costs of $1,000. We get one winner of $25k out of every ten tests, and six losers out of ten which cost $10k each. Residuals I’ll cover in a moment, but there aren’t any in the first month so it’s zero here. And Net Gain/Loss is the sum of the Wins and the Residuals minus the Fixed Costs and the Losses, for the month. Add all this together, and we get a $45,000 loss for the month.
Ugh! Wow, if all we did was one month of testing we’d surely think that Testing was costing us more than what we could ever get from it. But it that really the whole story?
As you might guess, it’s the Residuals that make the difference. If your team proposes and implements a test and it scratches or even loses money, you end up going back to your original version of whatever you tested. You incur the cost of the test but only during the time when the test is running. As soon as you determine the test was not a success you switch back to your original. You might do further testing in this same part of your site — or not — but that’s a different test.
But when you have a winner, you get to permanently run that version in place of your original, so even though the costs of the winner were paid up front, you get a “residual” impact, at no additional cost, to your efforts month after month into the future. I’ll repeat that: that is permanent, at least until you find an even better winner or underlying circumstances change. So Cato’s team is entitled to take credit for the improvement from the winner in month 1 across all future months when it considers the total impact of testing on the company going forward.
A Full Year Of Testing
Let’s push our model out to a full year of testing:
As you can see, in month 2, the Residual from month 1 kicks in, dropping the expected loss in half. By month 3, it’s wiped out the deficit and started to show a profit, because now we have Residuals from both months 1 and 2. And look at the numbers after 6, 9, and 12 months.
Now, is every testing month a linear multiple of previous ones? No, there will be variations. Some months you will not do as well; but other months you will do better than expected. The point is to keep on working hard and remain focused on what the long-term impact of testing has had on your company.
Penny Wise, And Pound Foolish
We’re almost done for today, though I promised you a short anecdote. This issue of properly attributing success via Residuals may seem obvious while you read this article, but I warn you it’s often forgotten “in the wild.” This loss of memory isn’t restricted to small companies.
I know one very large company who every reader would recognize by name (if I were free to say it), who started down the continuous testing path. After six months, the company was able to see direct, measurable improvement of several millions of dollars to their annual income based solely on the impact of their winning tests. Yet, they were “only” winning 1 out of 20 tests. And their costs of testing were running about $30k per month. What do you suppose was their next step? Get ready for it: they pulled the plug on their testing program. They couldn’t get past the fixed costs and the 19 out of 20 tests not being huge home runs. So don’t ever, ever doubt the ability of humans to snatch defeat from the jaws of victory.
That’s it for this part of this extended column on Testing. Tune in next time when we’ll dive deeper into how to measure your testing efforts. We’ll tighten up the model a bit, and discuss how to calculate the “true costs” of continuous improvement testing. I’ll also introduce the concept of the “opportunity cost” and “hindsight cost” of running a test, which is how you get senior management to deglaze and perk up when you show them your optimization team’s ongoing efforts.
Here’s the link to download the spreadsheet (XLS).
Comments