top of page
Writer's pictureFahad H

8 Ways We Deceive Ourselves With Metrics (And How To Avoid Them), Part I

“Oh, What A Tangled Web We Weave, When First We Practice To (Self) Deceive”


With apologies to Sir Walter Scott for my mangling of his most popular turn of phrase, I’d like to take some time now to discuss how people −and particularly people involved in metrics and evaluation of “how did we do this week/month/quarter?” − are prone to self-delusion.

Along the way, I run the risk of insulting you, but I see that as a necessary risk when confronting a topic like this. It takes a healthy sense of self-worth to say “holy crap! I’m a dufus!”

Now, let’s start off with some assumptions. First, that (most) people are honest, and don’t really intend to deceive themselves or others. (If you’re doing it intentionally, then bad karma on you, but I leave your fate to another writer.) This leaves with us two groups: the honest self-deceivers who do so consciously and those who do it unconsciously.

Conscious self-deception? Surely not!  Yet, how else do we explain common phrases for just this very thing:

  1. “I don’t want to know”

  2. “It’s got nothing to do with me”

  3. “Don’t make waves”

  4. “Look the other way”

  5. “Nothing I can do about it!”

  6. “Let sleeping dogs lie” (personally I love the double entendre here)

  7. “Ignorance is bliss”

  8. “Brush it under the carpet”

  9. “Why didn’t I listen to my intuition?”

All of these phrases are used in situations where the facts are uncomfortable and we consciously choose to ignore them. We make believe the situation is different, all the while knowing it isn’t so. I like to call this “quantum shifting” into an alternate universe where the unpleasant is no longer so.

Many of the situations where you’d use a phrase like the above are ones in which you feel you may have little or no control over things. So the natural human reaction is to ignore and hope that things change.

Is there a way to minimize self-delusion? Actually, yes, there is somewhat of a process you can use, though with varying levels of success.

  1. Admit that it occurs; realize, as a human, you will do it again

  2. Recognize when it’s happening

  3. Look for the opposite

  4. Friends don’t let friends self-delude: have a buddy who is willing to level with you

Again, I’d stress that the more compelling the bad news is, the more likely one is to steadfastly hold onto one’s convenient alternate reality. Let’s all group-self-deceive right now and chalk that up as “part of the charm of being human.”

Eight Common Self-Deceptions

Now, I can’t help you with that last buddy part. Nevertheless, here are eight common ways we unconsciously self-deceive using stats, ignorance, bad logic, or other wily human tricks. I’ll present them as encouragements − makes them funnier that way − so you can recognize when they might occur and be prepared to challenge yourself.

To encourage your own self-deception, it helps if you let yourself:

#1: Be Innumerate

Example: “what’s the big deal going from 2% conversion to 3%?? That’s only a 1% lift! We want more!”

Uh, no. Going from 2% to 3% is a 50% lift.

As the saying goes, “Innumeracy is a serious problem which affects 8 out of every 5 people.” I’m listing this as #1 on the self-delusion hit parade, because I see it the most often, and the people who practice innumeracy really do believe they “get math.” In some ways, you can argue this is easy to fix. In other ways, you can argue “you can’t fix stupid.” You decide which group you see yourself in.

#2: Start With your Wished-For Outcome… Then Look For Supporting Proof

Example: “Our Customers love us! We had 317 positive reviews on Yelp last month” (conveniently ignoring the 2,182 negative reviews and a poor Net-Promoter score)

I call this one the “Creationist Ostrich” technique. (Oh boy, I can see the comment box filling up now!) You start off in a comfy zone, and you really, really, really want to stay there, so you ignore all the evidence that contrasts with your view. Usually, this is because the alternative suggested by the evidence is bothersome in some way, and the more deeply bothersome, the easier it is to just dismiss, hold out with your head in the sand till 5 p.m., then go grab dinner. Problem solved!

Instead, under the part of the process mentioned above as “look for the opposite,” consider when you have a wished-for outcome that the best thing you can do for yourself is to try to find evidence against what you’d like the outcome to be. At least, if your goal is to uncover truth.

Virtually everyone involved in metrics, analytics, and testing-driven improvement online does, in fact, have a wished-for outcome: “get higher conversion,” “increase revenue,” “decrease cart abandonment.” Every metric you have for success implies you’d like more or less of that metric, so you always have a wished-for outcome at hand.

So, you’re subjecting yourself regularly to this sort of self-deception. Cut it off from the start by always trying to prove yourself wrong. Give yourself permission to be wrong, by constructing your tests in such a way that before you even run the test you can say, “heck if I see such and such from the test results, we know we’ve completely missed the mark” − and then look for such-and-such.

#3: Take Credit For Forces Beyond Your Control

Example: “Our sales were up in December!”

Yes, well, everyone‘s sales are up in December, especially in Retail. If you’re up by 20% yet everyone else is up by 30%, aren’t you actually down?

This self-deception, ranked #3 on my list, is a close relative of #1, but illustrates how you can get the math right, yet still self-delude. Commodity traders have a saying for this, by the way: “don’t confuse brains with a bull-market.” It’s very easy to make money − or in the cases of metrics and analytics, to increase the positive metrics you want more of − when market conditions are moving in the direction that gives rise to that in the first place.

The real artistry comes from those who do more with less. Current economic conditions are a great example of that. There are a lot of active testers and optimizers right now who are getting so-so lift from their experiments, but who are gaining valuable insight as to how to test and continuously improve.

Then there are those who are just going through the motions, dismissing key learnings, especially those that disagree with their pre-conceived notions or the corporate hoped-for storyline. Wanna make a guess as to which group is gonna look like all-stars when the next boom in their industry occurs?

#4 Settle for Industry Averages

Example: “We convert at 2.5% and industry average is 2.4%.  Yay us!”

A better measure is “how much did we leave on the table?” Or “how high could we get our conversion rate?” Or “what does the market leader do that we do not?”

This one can be hard to spot when you’re imposing this self-delusion on yourself. Often times, your industry average is the only other data you may have access to, due to proprietary info at your competitors. And the industry average is often comprised of self-reported stats, which do have a level of uncertainty or, shall we say, “hopefulness” or “putting your best foot forward” attached.

Nevertheless, do you consider your company akin to Garrison Keillor’s Lake Woebegone, where “all the women are strong, all the men are good looking, and all the children are above average”? Do you really want to be average? Don’t you want to excel?

What makes In-N-Out Burger customers so nuts about their fast food? What about the Apple “fan boys”? Do these companies sit around figuring out how to meet the industry average, or are they setting the industry average bar higher by striving for excellence above and beyond?

The biggest danger to this delusion #4 is when it combines with #3 as well as decent profit margins. For example, if industry averages for conversion rates were 2.5% and you are hitting a number seemingly much higher (let’s say, 5%) and you’ve got a very healthy profit margin. In such cases, almost everyone at the company will be patting themselves on the back, and the tendency will be to forget questions like “how much more could we really squeeze out of this?”

It’s in such situations that complacency sets in and eventually (who knows when?) some disruptive competitor comes along and completely blindsides you. Always, always, always be wary when your organization is doing too well, for that is the time you’ll most likely self-deceive with a combination of #4, #3 and #2. Celebrate the good times, yes, but keep a designated driver on hand.

More To Come

It looks like my space for this month’s column is filling up. Tune back in next time, when I’ll finish up with four more self-delusions on topics like Precision, Maxima, Causation… and Falsies. Yes, you too, can become expert self-deludatrix!

0 views0 comments

Comments


bottom of page