What gets measured gets done, no matter the cost. But there is a genuine danger in reducing product onboarding and design to an exercise in metric manipulation.
The last decade has brought us hundreds of analytics products. They all gather easy-to-collect data, and then render it in “beautiful to look at” visuals. Of course, no one checks if this “easy to collect” data is actually informative. Or if the “beautiful to look at” visuals are actually useful. But even when they are, and you have an accurate well defined funnel, playing whack-a-metric will inevitably lead you astray, blinded by funnel-vision.
Because what gets measured gets done. That phrase is often used to imply something like “Nothing will happen here until we have a clear metric” which is often true. Ambiguous phrases like “we’ll take a look at it”, “we’ll see if we can improve”, or “we’ll tackle the low hanging fruit” are the bane of any product team. It’s impossible to tell if anything actually happened as a result, whereas numbers don’t lie.
But numbers are selective with the the truth. When you instrument any flow, feature, or ratio, there will be a way to game it.
The Rat Effect
When Hanoi was under French colonial rule, they had a big rat problem. They tried to solve it by rewarding people for killing rats. To prove you had killed a rat, you presented it’s tail. What happened next is a good example of bad proxies. Citizens began to breed rats, cut their tails off, claimed the prize, and used the tail-less rat to breed more rats. The solution was backfiring. Yet at the same time the government was reporting “record progress” in rat extermination, as their metrics were through the roof. This became known as the Rat Effect, also known as the Cobra effect for a similar incident in India.
In both cases what got measured got done. And what got rewarded, got repeated.
All metrics, even if you hand pick them, are a proxy of user behaviour. Your project management tool might use “projects opened” to measure if users are getting value from the product, and that could well be a good proxy, but all proxies are leaky, and susceptible to gaming.
When we say “gaming” here, we mean influencing the metric in a way that doesn’t similarly influence what it’s a proxy for. For example, you could make “create a project” a mandatory step for every single user post-sign-up, or you could even do it automatically for them. The metrics will go up, but it’s not clear users are getting more value. You haven’t done anything substantial. It reminds me of the British sitcom where police officers boast they’ve reduced crime simply by legalising burglary.
Ev Williams uses “Total Time Reading” as a metric for Medium, but points out that there’s lots of ways to influence it that don’t help the cause, for example worse typography would easily be an “improvement” by that metric. Similarly I often joke that the editor Vim has the best growth hack of all-time, they boost daily active users by making quitting the editor ludicrously complex.
If you want to create meaningful growth your efforts are best spent in one of three ways, removing friction, relocating friction, or pointing users in the right direction. We’ve talked a lot about how pro-actively helping users increases conversion, decreases churn, and increases customer satisfaction, so let’s talk about friction. Specifically removing & relocating it.
Removing friction
The best metrics-driven meaningful growth will always come from identifying, removing, and relocating friction. If someone is at step 3 and fails to get to step 4, friction is usually what stops them.
Why did the user not get to step 4?
Discovery problem – They can’t see where to do it.
Clarity problem – They don’t understand what will happen if they do so.
Complexity problem – It’s too hard to complete the next step.
Value proposition problem – they’re not convinced it’s worth the investment of time/money
The point is we can tinker with red-vs-green button tests all day long, but any meaningful improvement is going to come from removing friction, not alternating shades.
Examples of removing friction
There was once a time when the only way to sign up for Intercom was to install JavaScript into your product and write some server side code to send us variables. That’s a lot of friction for uncommitted people. It also rules out a lot of folks who can’t easily re-deploy their own product. Our growth team removed this friction with a variety of approaches shown above, for example, our Stripe integration lets you start using Intercom in a couple of clicks.
Sometimes friction is best solved by accepting that some progress is better than no progress. For example, many users like to send HTML emails, but can’t, as they’d need a designer and developer to create them. So now Intercom users have access to a small bank of templates they can work with. Similarly some users want to set up some auto messages but don’t know what to say, so we’ve prepared some good examples to start with.
Moving friction around
A certain amount of friction you can remove, but the rest of it you move from place to place. This sounds like wasteful work, but it’s not. For any piece of friction that’s not removable, you try to find the area it does least damage, i.e. the area where it converts most people (not necessarily the area where it has the highest conversion rate though).
For example, steps like account registration, or asking for payment are nasty-but-necessary ones where you lose customers. Yes, you can simplify them using things like Facebook Login, or Stripe Checkout, but even so you’ll always see some drop-off once you ask for details or cash.
For example, offering a free trial, with no credit card required, increases homepage conversions, but you haven’t removed this friction, you just pay it back on day 30 instead. And in percentage terms on day 30 the credit card form will perform better than it did on day one. But again you haven’t removed any friction, you pay it back because less people see it. Similarly you can over-incentivise your users to convert, or start winning the wrong types of customers, but you’ll always pay it back in churn.
In short, reality always catches up, and the longer it takes the harder it hits.
Paired metrics
Lots of companies are adding growth teams at the moment, and it’s likely to increase over the coming years as their true value becomes clear. If you’re adding a growth team it’s worth taking Andy Grove’s advice on metrics and measurement: for every metric, there should be a paired metric that addresses adverse consequences of the first metric.
If nothing else remember Nietzsche’s statement: The most basic form of human stupidity is forgetting what we are trying to accomplish.
Comments