In 1994, the Robot Exclusion Protocol was established as a voluntary system for webmasters to control which robots could access which pages of a website. In a nutshell, under the system, robots such as Googlebot announce themselves to Web servers and download a file, robots.txt, that says which pages are allowed and not allowed to be crawled.
When there have been disputes between search engines and publishers, the search engines can point to this protocol and say that publishers are free to block crawling, indexing and caching of their Web pages. This voluntary system has largely worked.
A similar voluntary protocol for ad blocking should be established. It would be far better for publishers and tech companies to come up with an agreement about the nuances of ad blocking, rather than pummel each other in court.
Moreover, by establishing industry standards, rogue operators who disregard the norms can be excluded, leading to a better experience for users and more profit for publishers.
To design an ad-blocking protocol, we first need to understand what each side wants, so that the protocol will be fair and beneficial to all concerned.
Users have reasons to use ad blocking:
Users pay for bandwidth. They may not want to pay for a multiple-megabyte download when they unknowingly view a page that is overloaded with ads and tracking scripts.
Users have a right to privacy. They may not want to have their activities tracked before they have had a chance to read the site’s privacy policy and give their consent to be tracked.
Users do not want their browsers to crash or become infected with malware distributed by a dodgy ad network.
Publishers have legitimate concerns about ad blocking:
Publishers may need to monetize their content with ads to stay in business.
Publishers may want to track user activity to improve their content, to serve relevant ads and to accurately charge advertisers.
Publishers have the right to set conditions for accessing their websites. For instance, most publishers forbid republishing of their Web content without permission.
The Proposed Protocol
Ethical ad blockers should identify themselves to Web servers in the http request headers. Publishers should be able to set access policies, including a requirement that users whitelist the site, so that ads are not blocked, as a condition for accessing content.
On the other hand, publishers should be obligated to accurately disclose to users what sort of ad formats will load (text, banner, interstitial, auto-roll video) and the maximum bandwidth consumed by ads (a maximum number of kilobytes per page view), and they should provide a link to the website’s privacy policy.
Users ought to be able to learn what’s going to happen before loading the first page onto their computers through their internet connections. These disclosures would give publishers an incentive to maintain reasonable ad payloads and privacy policies to avoid losing visitors.
An Example Session
The transaction between browser and server would go something like this:
Browser: Hello, I’m a Chrome browser running on Windows 7 with ad blocking enabled. I would like to view your Web page.
Server: Acknowledge your request. Before I grant access, you should know that this content is sponsored by ads, and we require visitors to whitelist our site as a condition of access. If you choose to load our pages, you may receive text and banner ads. The bandwidth consumed by ads is never more than 800 KB per page view. You should know that we use cookies to track user access to our site. Our full privacy policy can be found here (link). Do you agree with these terms?
At this point, the browser displays that info to the user and requests him or her to choose whether to whitelist the site so that ads can be shown or to reject the terms.
For the sake of efficiency, ad-blocking companies can pre-load whitelists of sites known to have well-behaved ads and reasonable privacy policies. Charging for placement on such a list would not be very different from the way the Better Business Bureau charges for membership or various online trust certifications charge for their security seals.
Somebody has to review the site and make a determination, and that entity is entitled to a reasonable fee for the value provided.
How To Keep Everybody Honest
The way to ensure that publishers make honest disclosures is for ad-blocking companies and anti-abuse organizations to maintain blacklists. Just as there are blacklists of email spammers, there can be blacklists of publishers who fail to accurately disclose their advertising payloads and privacy policies.
If a complaint is filed, publishers can be warned and given a reasonable amount of time to correct any inadvertent errors before they get blacklisted.
On the user side, ad-blocking browsers and plugins should disclose themselves to publishers. This disclosure would be voluntary for ethical ad-blocking products.
Those products that choose not to disclose could lead a user to violate publishers’ website terms of service and potentially create legal liability.
I doubt that publishers would sue users for running undisclosed ad blockers, but it seems possible, or even likely, that publishers would sue the providers and distributors of stealthy ad blockers. By offering a reasonable alternative, namely a protocol for ethical ad blocking, publishers would strengthen their position against rogue ad blocking.
The digital advertising industry needs to avoid an ad-blocking arms race where publishers try to circumvent blocking and ad blockers try to remain undiscovered.
The industry also needs to put forward a reasonable alternative that recognizes the nuanced reasons why users might use ad blocking and gives publishers an incentive for good behavior.
Comments