Back when Google was an upstart search engine, one way it distinguished itself was to fight against a pay-to-play business model called “paid inclusion.” Indeed, paid inclusion was one of the original sins Google listed as part of its “Don’t Be Evil” creed. But these days, Google seems comfortable with paid inclusion, raising potential concerns for publishers and searchers alike.
What Is Paid Inclusion?
There are two main ways that companies show up in Google’s search results. Companies either buy ads or hope that Google decides that their content is relevant enough to appear in what are called the “organic” or “natural” or “editorial” listings.
Companies can’t pay Google to show up in these organic results. It’s similar to how things work with a reputable newspaper. Companies get “coverage” to the degree they are deemed to have earned it. The only difference is that Google uses a search “algorithm” that weighs various ranking factors to decide what’s relevant, rather than employing human editors and reporters.
Here’s another similarity with newspapers. Most people ignore Google’s ads, just as most people ignore newspaper ads. Still, enough people click on those ads to make Google billions of dollars per year. But imagine how much more Google might earn if it could somehow sell those main listings, the editorial ones?
That’s where paid inclusion comes in. It’s a method that search engines (other than Google) used to use sell those main listings yet still claim that there was an “editorial” component to them. With paid inclusion, companies pay to be considered but aren’t guaranteed to rank well. The algorithm ultimately decides, not payment.
Google, Evil & The Death Of Paid Inclusion
If all that makes your head hurt, you’re to be forgiven. Believe me, years of trying to explain how paid inclusion works made my own head hurt.
Search engines like AltaVista or Inktomi would tell the searching public that they had the freshest, most comprehensive collection of web pages out there. These same search engines would tell publishers that they should do paid inclusion because otherwise they might miss collecting their pages or take ages updating them. Talk about mixed messages!
Google was the main holdout among the major search engines. Back in 2001, when paid inclusion programs were growing in popularity, it told me:
We have no plans for a paid inclusion program. As we’ve stated in the past, our search results represent our editorial integrity, and we have no plans to alter our automated process, which works very well in gathering information and delivering highly relevant results.
When Google went public in 2004, the Founders Letter that was part of its IPO filing specifically named paid inclusion as a practice that should be shunned, saying under the “Don’t Be Evil” section:
Google users trust our systems to help them with important decisions: medical, financial and many others. Our search results are the best we know how to produce. They are unbiased and objective, and we do not accept payment for them or for inclusion or more frequent updating.
The full IPO filing has other references against paid inclusion, such as (I’ve bolded the key parts):
Objectivity. We believe it is very important that the results users get from Google are produced with only their interests in mind. We do not accept money for search result ranking or inclusion. We do accept fees for advertising, but it does not influence how we generate our search results. The advertising is clearly marked and separated. This is similar to a newspaper, where the articles are independent of the advertising. Some of our competitors charge web sites for inclusion in their indices or for more frequent updating of pages. Inclusion and frequent updating in our index are open to all sites free of charge. We apply these principles to each of our products and services. We believe it is important for users to have access to the best available information and research, not just the information that someone pays for them to see.
And:
Froogle [now called Google Shopping] enables people to easily find products for sale online…. Most online merchants are also automatically included in Froogle’s index of shopping sites. Because we do not charge merchants for inclusion in Froogle, our users can browse product categories or conduct product searches with confidence that the results we provide are relevant and unbiased.
After Google went public, Microsoft and Ask.com, feeling the pressure, dropped their paid inclusion programs later that year. Yahoo was the last holdout, finally dropping its program in 2009.
Google & The Resurrection Of Paid Inclusion
Having previously declared paid inclusion to be evil, can it really be that Google is doing it now? Yes, though Google’s not been using that name, and it also really didn’t become apparent until last month.
Google Hotel Finder and Google Flight Search were both launched last year. On the surface, they seem like Google’s other “vertical” or topic-focused search engines, such as Google Images, Google News or Google Shopping, which allow you to search for certain specific types of information.
Unlike Google’s other vertical search engines, however, payment seems to have a role in being included in Google Hotel Finder and Google Flight Search. Maybe Google will find your hotel or your flight and list it for “free.” But it seems more likely you’ll appear if you have a paid relationship.
This only became evident after Google launched its new comparison boxes last month, as we asked questions about how the underlying services that power those boxes got their results.
The boxes also draw from Google Advisor, which is a type of financial products vertical search engine. Listings for credit cards, CDs, checking and savings accounts can be found with Google Advisor, but mainly from merchants who pay to be included, it seems.
This is a dramatic change. How does Google justify the shift? When I spoke with Amit Singhal, the Google executive who oversees all of Google’s search products, earlier this month at our SMX London conference, he explained that Google needed to have paid relationships to gain some data:
Fundamentally, time and time again, we started noticing that a class of queries could not be answered based upon just crawled data…. We realized that we will have to either license data or go out and establish relationships with data providers.
Since some of this data required paid relationships, Google decided some type of disclosure was required:
To be super safe, where we have a deal between Google and another party, we didn’t want to call those fully organic results, because they are based on a deal…. After much debate, we said “OK, let’s be extra cautious. Let’s call it ‘sponsored’ so that we tell our users that there’s a special relationship that Google has established with someone.”
But what about cases where Google has traditionally been able to get data for free, such as with shopping search? Might that or other verticals shift eventually to a paid inclusion model? Singhal said:
In every area we’re looking at what’s the best possible model…. Clearly all these areas are ripe for innovation, and that’s what we’re going to do.
You can watch our full discussion in the video below:
Disclosure & Paid Inclusion
The return of paid inclusion makes for some murky waters, for both searchers and publishers.
For searchers, it becomes harder to know if you’re really getting comprehensive information or not. The US Federal Trade Commission’s guidelines on paid inclusion disclosure aren’t as clear-cut as with ads. They require that search engines have some type of “clear and conspicuous” disclosure but not that the paid inclusion listings be segregated from unpaid listings, as is the requirement for ads.
The disclosures, the FTC declared back in 2002, should allow consumers to:
Easily locate a search engine’s explanation of any paid inclusion program, and discern the impact of paid inclusion on search results lists. In this way, consumers will be in a better position to determine whether the practice of paid inclusion is important to them in their choice of the search engines they use.
At Google, disclosures within the new comparison boxes are easily located, through a small “Sponsored” link at the top of the boxes. These boxes all appear within the regular search results at Google, when deemed relevant to hotel, flight or financial product searches.
Here’s an example with flight results, where I’ve hovered over the Sponsored link to make the disclosure appear:
Here is it is for hotel search:
Here it is for a financial product search:
The disclosures are easily located, but they don’t lead to deeper explanations that would better fulfill the FTC’s guidelines. What percentage of listings come from paid inclusion with these services, for example. Are only those who pay listed? These are questions consumers might wish to discover.
The situation is worse if you go directly to any of the services that provide results to the comparison boxes. For example, Google Hotel Finder has no disclosures. In fact, the help page suggests that all listings are free. If that’s true, then why would Google be disclosing a financial relationship for when Google Hotel Finder results appear within a comparison box?
Google Flight Search has no disclosures that I can locate, nor does its hard-to-find help page explain about inclusion policies. As for Google Advisor, the home page says that Google is not being paid for any of its listings despite the fact that Google told us last month that it is and that the text would be updated.
Of course, it’s not uncommon for me to go to vertical search engines that seem to lack any disclosure at all. The FTC hasn’t seemed to care about enforcing its paid inclusion guidelines, but with paid inclusion on the return with Google, maybe it will wake up.
End Of The Free Ride?
For publishers, the key issue is whether they’re going to be charged in the future for traffic from Google that currently is considered “free.”
Remember all those news publishers a few years ago that viewed Google News as somehow robbing them of content? The ones that got all that free traffic yet still felt that Google should pay them to be included in Google News?
With paid inclusion, it’s possible that Google could turn the tables. You want to be included? You pay. No pay? No play.
End Of Don’t Be Evil?
Having previously declared paid inclusion to be evil, is Google now evil to be doing it? The reality is that paid inclusion as a concept wasn’t necessarily evil or even bad, any more than traditional yellow pages — effectively a print-based form of paid inclusion — are bad. Even the FTC found good reasons for paid inclusion to exist.
I never liked paid inclusion when search engines were using it as an revenue-generating excuse to do things they should just ordinarily do, such as gather content up quickly and keep it fresh. It was especially appalling when it effectively turned into an approved cloaking system for Yahoo.
But paid inclusion does make more sense in the vertical space and perhaps also in a world that’s nearly 10 years and 100 internet dog years removed from when paid inclusion was in its heyday.
Ironically, the change might hurt Google in its battle against anti-trust regulators. I wrote before how odd it was last year to have a search engine like NexTag — which seems to be a pure paid inclusion service — arguing that it needed more visibility in Google, since that ironically might harm far more businesses rather than help them:
When people go into Google Shopping, they find listings from thousands of merchants all across the web, merchants that pay exactly $0 to be listed in the editorial listings…. The committee … oddly seems more motivated to ensure that NexTag pays less in advertising than the fact that regulating NexTag’s inclusion in Google might cause thousands of individual merchants to pay more.
If Google Shopping really was taking away NexTag traffic, at least the thousands of companies within Google Shopping were in turn receiving traffic from Google for free. But if Google Shopping were to move to paid inclusion, there’s one argument in Google’s favor that gets weakened.
As I said, paid inclusion isn’t necessarily bad, especially if it’s used to solve an otherwise difficult challenge in search, rather than being an excuse to generate revenue. However, it it still feels odd watching Google, having previously attacked the objectivity of its competitors over the practice, quietly adopt paid inclusion now that it’s the search market leader. That doesn’t sit right. At the very least, I kind of want someone at Google to acknowledge that it was wrong those years ago.
Postscript (7:30pm ET): Google, after seeing this article, sent along this statement about paid inclusion:
Paid inclusion has historically been used to describe results that the website owner paid to place, but which were not labelled differently from organic search results. We are making it very clear to users that there is a difference between these results for which Google may be compensated by the providers, and our organic search results.
I have to disagree. Paid inclusion has been historically used to describe payment to appear in search listings but without a guarantee of prominent placement. That’s exactly why we have the term “paid placement” that was used those years ago versus “paid inclusion.”
Perhaps the best definition of paid inclusion is the one that the FTC put forth in 2002, since it made that definition specifically to advise search engines on how they should disclose paid inclusion. The definition:
Paid inclusion can take many forms. Examples of paid inclusion include programs where the only sites listed are those that have paid; where paid sites are intermingled among non-paid sites; and where companies pay to have their Web sites or URLs reviewed more quickly, or for more frequent spidering of their Web sites or URLs, or for the review or inclusion of deeper levels of their Web sites, than is the case with non-paid sites.
The definition says nothing about a lack of disclosure as being a defining characteristic of paid inclusion. Moreover, when Google was opposed to paid inclusion all those years ago, I did not see it using the current definition it’s offering now.
The bottom line is that Google is doing very good disclosure of its current paid inclusion implementations. There’s room for improvement, but it’s far better than how many of its competitors disclose paid inclusion, if they do at all.
But disclosure doesn’t mean that Google is not doing paid inclusion, nor that Google wasn’t opposed to it in the past. It did characterize paid inclusion as some type of evil back then to be avoided; it clearly no longer views paid inclusion as evil but rather helpful in the search process.
Postscript: Google Product Search To Become Google Shopping, Use Pay-To-Play Model on our sister-site Search Engine Land covers how Google is now shifting its shopping search engine to a pure paid inclusion model. It’s the first time Google’s changed a service that had free listings to being exclusively paid.
Comments