OpenAI, Google, Microsoft and Anthropic to adopt a shared signal for opting out of AI scraping
The four major AI companies say they will support a new standard that lets website publishers opt out of AI training across all of them at once. The signal will be placed in the robots.txt file.
AI companies OpenAI, Google, Microsoft and Anthropic have announced they will adopt a new standard that website publishers can use to opt out of having their content scraped for AI purposes. Until now, publishers have had to state their AI training preferences separately to each company; the new approach lets them do it once for all. The plans were first reported by Search Engine Roundtable.
How it will work
The shared “policy token” signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF). It will sit in a site's robots.txt file, which has long indicated which web crawlers may or may not scan a site, whether for search indexing, archiving or AI training.
OpenAI, Google, Microsoft and Anthropic say they will keep their AI crawlers away from a site if they detect the shared token. Anthropic has already adopted the signal, while Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”
Extra options from Microsoft and Google
Microsoft says it will provide more granular controls allowing publishers to opt out of AI training or of AI grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations. The company adds that websites will still appear in Bing search results regardless of their settings. Google has previously introduced similar controls that let sites decide whether their content appears in AI search features or is used for AI training, without affecting search visibility.
Background
The announcement comes as publishers contend with a continuing flow of AI crawlers scraping their sites and feeding the content into AI models. Others have already tried to counter unauthorized scraping: the Really Simple Licensing (RSL) standard lets publishers charge crawlers that visit their sites, and Cloudflare blocks AI crawlers by default.
Meta communications head Andy Stone also voiced support for the new setting but did not say whether the company will honor it. He said that creating a common signal that conveys such preferences to multiple providers at once is an important step, and that Meta looks forward to working with other AI companies on the standards and their implementation.
