You already know about Facebook over-calculating how much time people spent, on average, watching Pages’ organic videos and brands’ video ads. You already know it’s done away with only calculating that figure when people viewed a video for at least three seconds. But how much do you know about how Facebook is replacing that metric, or whether the platform-provided metric even matters?
Facebook’s new average watch time metric may sound straightforward, but it has a twist. And soon enough, it won’t be the only average watch time metric that Facebook offers advertisers and Page owners.
Facebook’s new average watch time measurement, which is provided for Pages’ organic videos and for video ads, takes the total amount of time that people watched a video and divides it by the number of times that video was played, either automatically or after being clicked. Simple enough, except that Facebook defines a “play” differently from the way you might expect. Facebook counts a play only once when someone watches a given video within a 30-minute period that the company refers to as a “user session,” i.e., any replays within that time frame are not counted, and it’s possible for a video’s average watch time to exceed the actual length of the video, a Facebook spokesperson confirmed.
Let’s take a 30-second cooking video as a hypothetical example of how the new average watch time metric is calculated.
I come across this video while scrolling through my news feed, but I’m not paying attention to it yet. I scroll slightly past it to some post about some back-to-back-to-back shark attacks in Florida, so it plays in view but only partially. After 15 seconds, a splash of milk catches my eye, and I watch the rest of the way through. It’s a cooking video, so I’m mesmerized. But I don’t catch the first step, so I replay it but only to see the 15 seconds I missed in the first watch. As measured by Facebook, I spent 45 seconds watching the video but only played it once, since the replay happened during the same user session as the initial play.
My girlfriend also comes across the video. She’s cool with cooking videos but is lactose-intolerant, so when the milk gets poured into the frame 15 seconds in, she tunes out. Facebook jots this down as 15 seconds in watch time and one play.
Then Facebook crunches the numbers. Based on these two viewers alone, the 30-second video’s average watch time clocks in at 30 seconds. And its percentage watched metric — a video’s average watch time divided by its actual length — is 100 percent, even though the video was only watched to completion once.
Facebook’s new average watch time (and percentage watched) metric “is telling you something, but I would chalk it up more to being confusing than necessarily good or bad,” said Noah Mallin, head of social at MEC North America.
It is confusing. Probably because we’re used to thinking of average watch time as a measure how long people spend watching a video each time they watch it. But that’s not what Facebook is measuring here. This isn’t average watch time per view; it’s closer to average watch time per viewer (though if it were really per viewer, then Facebook would count one play per user, not one play per 30-minute user session, and thereby count replays outside of that user session as if the viewer were somebody else entirely). Facebook is trying to account for people who watch a video more than once because, according to a Facebook spokesperson, advertisers have told Facebook they want to factor in repeat viewers because that would help them evaluate a video’s popularity.
“If average watch time exceeds the length of the video, that means you have something incredibly watchable,” said Mallin.
That watchability can also translate into the numbers that advertisers really care about, like whether someone is more likely to think of a brand when they’re in the market for a product in its category. That’s why Facebook’s methodology blunder — “[T]hey had the wrong denominator,” said Horizon Media’s executive VP and managing partner of digital investment, Sarah Baehr — has gotten so much attention. On the one hand, it’s a math error specific to a supplemental metric that doesn’t affect how brands are charged for their ads; on the other hand, it’s a metric that has a ripple effect.
“Time spent with ads and content is often highly correlated to metrics like awareness, interest and action like sales, so [average watch time] is an important KPI [key performance indicator],” said Scott Symonds, managing director of media at AKQA.
What about brands, publishers or other Page owners who want something closer to the old, if ill-defined, average watch time metric that told them how much time people spent watching a video per view? Facebook’s working on one.
Facebook is developing a Unique Average Watch Time metric (and corresponding Unique Percentage Watched metric) that would approximate a video’s average watch time by view. According to a Facebook spokesperson, the unique average watch time would take the longest single instance that an individual watched a video and add that duration — and only that duration — to the video’s total watch time, count one play toward the total number of plays and divide total watch time by total number of plays.
Let’s return to that hypothetical 30-second cooking video to lay out how Facebook’s unique average watch time measurement is calculated.
I watched the video twice — once in full and again for 15 seconds — and my girlfriend watched it once, but only for 15 seconds. Instead of adding both of my watch times, Facebook would only take the longest one because it just wants to know the maximum amount of the video I watched in one go. So my 30 seconds plus my girlfriend’s 15 seconds equals 45 seconds in total watch time. Since Facebook is only taking one watch instance per viewer, it’s taking one, and only one, play per person. So while the normal, non-unique average watch time totaled 30 seconds (100 percent watched), the unique average watch time would work out to be 22.5 seconds (75 percent unique percent watched).
Neither Facebook’s new average watch time or upcoming unique average watch time metrics are perfect. But even if they were more exact — such as by removing the 30-minute user session stipulation to make average watch time a truly per-viewer metric — they wouldn’t be perfect.
“In a perfect world, what we want to see is complete third-party verification of all data,” said Mallin. “That’s what everybody would like, and it’s not just a Facebook issue.”
It’s not necessarily that media buyers think Facebook would fudge its numbers to make itself look better — or if they think that, they don’t say it aloud, at least not to me — but that an independent auditor could erase room for error. Publicis Groupe used Facebook’s measurement error to emphasize “the absolute need to have third party tagging and verification on Facebook’s platform,” the agency holding company wrote in a letter to clients, according to The Wall Street Journal.
But here’s the thing about that need for independent verification of Facebook’s video ad metrics: It’s already available. In the past year, Facebook has signed deals with Moat, comScore, Integral Ad Science and Nielsen to independently measure its video ads. And Moat CEO Jonah Goodhart told AdExchanger earlier this week that his company’s measurements of Facebook’s video ads, including average watch time, “are accurate.”
And it’s not like measurements from independent firms are immune to error. Earlier this year, comScore got called out for over-calculating how much time people were spending within Facebook’s mobile apps. The reason? A bug in Facebook’s code that made it seem like people were actively checking Facebook when in reality the app was running in the background.
But the fallibility of Facebook’s or independent firms’ measurements isn’t really the issue. They’re a problem part of a bigger problem. Many brands want to be able to measure their Facebook ads in a way that they can stack them against their Snapchat ads or Twitter ads or YouTube ads or TV ads. That’s the root of why they want to have objective parties do the measuring; those firms can set their own measurement methodologies in order to establish metrics that can be applied across different platforms. “We prefer to use third-party metrics so they are comparable across platforms,” said Symonds.
So would sports fans trying to settle barstool arguments over who’s the greater athlete among Serena Williams, Michael Jordan and Michael Phelps. But what are the criteria: Championships won? Championship winning percentage? Quality of opponents? Longevity? Difficulty of sport?
Not all media buyers are in agreement that, even if such apples-to-apples comparisons become available, they’re worthwhile.
“The key thing is how do we get to a metric that’s about evaluating video that’s on a mobile phone within a feed,” said Baehr. “And if it really is about a replacement for a declining viewership in TV, is the expectation that that video is one-for-one going to have the same impact? I think we all agree that watching anything on your mobile phone is just different.”
And even within the context of the mobile phone, it’s different. A sound-on, full-screen vertical video on Snapchat is not the same as a sound-off, in-feed horizontal video on Facebook is not the same as a sound-on, full-screen horizontal video on YouTube.
At the risk of overusing sports analogies, it’s like trying to compare athletes of the same sports across eras. It’s easier to compare Serena Williams to Steffi Graf and Billie Jean King than to Michael Jordan and Michael Phelps — but not by much. You can standardize the comparison by looking at stats like matches won and winning percentage. But how do you account for less quantifiable variables. Does it matter that medical treatment and training/diet regimens were different for each athlete, and if so, by how much? What about how much money they made in their careers, and the extent to which that income relieved (or didn’t) off-court pressures, making it easier or more difficult to skip less meaningful tournaments or smaller obligations? What about travel accommodations? Or the pressure that comes with how high-profile the sport in general, and Serena in particular, has become? How do you account for context?
But let’s say marketers come up with a way to quantify context. Maybe there’s a way to weigh sound-on versus sound-off, vertical vs. horizontal, full-screen vs. in-feed. Then the concrete metrics become even more important.
If a brand can know with some certainty that a person watching a sound-on, full-screen vertical video is worth 150 percent more than a sound-off, in-feed horizontal video, then it matters if people are spending 150 percent more minutes on average watching the sound-off, in-feed horizontal video. And it really matters that those time-spent measurements are accurate, not inflated by 60 to 80 percent, as Publicis has claimed Facebook’s self-reported average watch time was.
Without those concrete metrics as a solid foundation, any consideration of context and any cross-context evaluation crumbles.
“Everybody wants to understand better what the actual impact is of what they’re doing, not just on Facebook but all of these platforms,” said Mallin. “Since they’re such different environments, it lends itself to having a lot of different questions arise. The more we’ve been able to learn from the platforms, the more ability we have to put things into context and get down to true performance.”
Comments