OpenAI and Microsoft said on Friday, Oct. 30, that they will honor one shared signal websites can use to keep their pages out of AI training, and Google said it plans to. Anthropic, one of whose employees wrote the proposal, says it has started building support too. The signal goes in robots.txt, a plain text file on a website that tells crawlers which pages they may visit. Crawlers are the programs that visit web pages automatically. The new signal does not stop them from visiting; it states how the site wants its pages used.
Until now, a site that wanted to stay out of AI training had to name each company's crawler and keep separate rules for every one, OpenAI says. Under the proposal, a site writes its rules once under one shared label, __usage_ai-training, which goes where a single crawler's name would normally go, and every company that honors the signal reads the same answer. A site can opt out for all its pages or only some of them.
OpenAI, Google and Microsoft don't honor the signal yet. OpenAI aims to start in the coming months, Google aims to support it in 2027, and Microsoft says it will honor it from Feb. 1, 2027, as its crawler reaches each page. Anthropic gave no firm date and says adding support across its services will take time. OpenAI, Google and Anthropic say their own company settings keep working alongside it, such as Google-Extended, the setting Google has offered since 2023 to keep a site's pages out of its AI training. Google says that if a site uses the shared signal to opt out and sets no Google-Extended rule, it plans to count that as an opt-out through Google-Extended too.
Microsoft is also adding a setting of its own, for whether a page may be used to help shape the answers its AI gives, including in Bing, Copilot and Microsoft 365, and it warns that blocking this use will make a site much less visible in its AI answers. Microsoft will stop reading its older opt-out tags after March 1, 2027, so sites that use them must set the new preferences by then to keep their choices.
The proposal is one author's draft at the Internet Engineering Task Force, the group that published the standard for robots.txt, and the group itself has not endorsed it. Following it is voluntary, and the signal covers training only. A site that writes nothing has stated no preference, and Microsoft treats that as permission to train on its pages. Microsoft says the settings do not set up licensing or payment and guarantee no traffic. According to a news report, Meta voiced support but did not say whether it would honor the signal.