Most marketers are well-acquainted with the
concepts of A/B and multivariate testing, but for many organizations, getting involved in testing still represents stepping into a new frontier.
In recent years, a variety of tools have made testing much easier, and therefore more attainable for more organizations. That’s a great thing — the ability to test, learn and optimize shouldn’t be restricted to only the biggest, richest or most technically savvy.
But if you’re thinking about getting started — or getting more serious — with testing, you’ll find there’s more to success than simply acquiring a testing tool and launching as many tests as possible. Instead, to maximize your chances of seeing long-term ROI (return on investment) on your testing efforts, start by building a strategy for optimization.
Testing is exciting for a lot of organizations because it represents the ability to go beyond simply collecting and reporting on data. It’s an opportunity to activate your data by using it to generate specific hypotheses, test those hypotheses and reap the benefits.
But where to begin? There’s so much you can do with testing that without a well-defined strategy, the path forward likely won’t be clear.
Zero In On Your Top Strategic Goals
When I build a testing plan, I try to begin by focusing on an organization’s top strategic goals. On the surface, this might seem a bit silly — for example, most companies want to increase revenue and improve profitability.
Fair enough, but a testing strategy that prioritizes “growing revenue” isn’t very specific. How exactly will you grow revenue, and why will you choose those areas of focus?
A revenue-focused strategy for an e-commerce organization could prioritize any of the following (and more):
Increasing purchase frequency of existing customers.
Converting more visitors into first-time buyers.
Improving performance of upsell/cross-sell.
Testing price sensitivity.
An understanding of your business model, coupled with quantitative and qualitative analysis, will help you build a strategy that focuses your efforts in specific areas. And that’s no small thing.
Your strategy will naturally influence the kinds of tests you design and launch, but before you even get there, it will influence the tools you’ll need to be able to execute.
Decide How The Testing Process Should Work
After hashing out what exactly we’re hoping to accomplish by testing, I like to focus on putting together a framework for how the testing process should work. A lack of governance of that process can undermine your strategy and lead to missed opportunities.
Indeed, I often find this happening with clients who have already been testing for a while. One of our clients in the .edu space had been running tests for a long time, but to little effect. As you can imagine, this was starting to undermine confidence in testing and was causing stakeholders to question whether testing was worth the continued effort.
What we discovered was that this university had a clear testing strategy — such that tests were consistently intended to generate lift against previously agreed-upon metrics — but was suffering setbacks due to process and governance issues. Examples of issues included:
Running tests that yielded no results due to changes not being significant.
Running tests that would never end due to too many variations being used.
Stopping tests before results could be conclusive/reliable.
Fortunately, these issues can be cleared up fairly quickly — but it’s much easier to avoid them entirely by addressing them up front.
For example, as you start thinking about some specific tests you’d like to run, consider the following.
How big is your audience, and what does that mean for the kinds of tests you can run?
How many variations can you realistically test in a reasonable time frame?
How conservative — or aggressive — will your changes need to be in order to yield conclusive results?
About how long will your test have to run before you’ll have conclusive results?
Doing this kind of research up front allows you to take your strategy one step further. Not only are you clear and focused on what you’re trying to optimize toward — you’re going in with “eyes wide open” in terms of the kinds of tests that can you help you yield results in a reasonable time frame.
Note that putting these guidelines in place doesn’t mean you’re stifling the very creativity you’re trying to take advantage of in your testing program; it means that you’re focusing your team’s efforts on creating tests that have the best chance of success.
For example, imagine a scenario in which an organization is trying to optimize its home page’s information architecture in order to help users consume content deep within the site. Depending on how much traffic that home page receives, certain kinds of tests will — or won’t — be feasible.
For a site receiving a lot of traffic, a multivariate experiment using a large number of variations could be a great option. Conversely, for a site with a smaller audience, it may be more appropriate to start with a simple A/B test and iterate based on learnings from initial results.
Don’t Forget To Stay Flexible
No matter what your goals for testing and optimization, one more crucial ingredient in your strategy is flexibility. Many testing strategies ultimately rest on assumptions that we make about our organizations. Over time, you may find that some of those assumptions are unsound.
Recently, one of my clients had to rework their optimization strategy because a fairly simple assumption — that maximizing the number of leads generated online will maximize the number of paying customers offline — turned out to be unreliable. As a result, we’re shifting our focus to designing experiments that use what we know about the most qualified leads.
This means an increasing focus on personalization — keying in on the most valuable specific segments and optimizing the experience for them.
As you embark down the path toward optimization, it’s probable that you’ll eventually hit a “bump in the road” that requires you to go back to the drawing board. Nobody can promise that every test will yield a winner or that your original testing strategy will always make sense.
But by developing a strategy in the first place, ensuring that the tests you run are consistent with that strategy, and being willing to adapt over time, you’ll position your organization to realize the long-term potential of optimization.
Comments