Cloudflare has introduced new controls that let website owners decide separately how their content can be used for search, AI training, and AI agents. The move aims to address the conflict between staying discoverable online and preventing content from being used to train AI models.
The company introduced the ‘Accountable’ designation for AI crawlers, providing websites with a clear way to control AI training and maintain search visibility.
“This is how we make the Internet better: preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used,” said Matthew Prince, co- founder and CEO of Cloudflare.
Controls amid crawling
Cloudflare is also replacing its single “Block AI Bots” switch with three independent controls covering search, AI training and AI agents. It will also replace Managed Robots.txt with Bot Preference Sync, letting website owners set preferences once and automatically apply them across supported crawlers.
The company has established four requirements for crawlers to receive its Accountable designation:
- Giving site owners a clear way to opt out of AI training through robots.txt or a comparable standard.
- Allowing site owners to opt out of AI-generated search summaries
- Providing site owners with URL-level visibility into how their content is used for search and training
- Publicly confirming that opting out of training will not affect a site’s ranking in traditional search
Cloudflare said Apple, Google and Microsoft have met or committed to meeting these requirements. Google, for example, offers Google-Extended, which allows websites to opt out of training without opting out of search.
The company is also working on AI content preferences standards through the IETF, including developing the “ai-prefs” specification, which would allow any website to express its AI access preferences in a standardised, portable way.
