The robots.txt fix for blocking AI training without losing search
Cloudflare's newest crawler setting solves a false choice, block AI training or stay visible in AI search. Google and OpenAI already let any site do the same, free.
Cloudflare added a new setting to its dashboard on September 15: block an AI company from training its models on a site's content, without also blocking that site from Google Search or ChatGPT's search results. Until now, a Cloudflare customer had no automated way to say yes to one and no to the other for a class of crawler Cloudflare calls "mixed use," where the same bot indexes a page for search and feeds it into a training pipeline. The dashboard's blunt "Block" control skipped those crawlers entirely rather than risk taking a site out of search. The fix is not exclusive to Cloudflare's own customers, either. Google and OpenAI already publish the exact robots.txt lines that do the same job, for any site, at no cost.
Google already separates training from search in its own documentation
Google's crawler documentation names two things for what is often the same request. Googlebot crawls a page for Search and the products built on it. Google-Extended is a second name a site can address in robots.txt, used only to control whether that same content trains future Gemini models or grounds Gemini Apps and Vertex AI. Google states the boundary plainly: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." A site that disallows Google-Extended and leaves Googlebot alone keeps its search ranking and opts out of training, in one robots.txt block. Google-Extended has no crawl of its own to send; the token governs what happens to a fetch that is already being made under another name, not a second bot arriving at the door.
OpenAI splits the same decision three ways, and only two of them are controllable
OpenAI names three agents for what a single vendor's crawling might otherwise be read as one job. GPTBot collects "content that may be used in training our generative AI foundation models," and disallowing it opts a site out of that use alone. OAI-SearchBot is the one that "surface[s] websites in search results in ChatGPT's search features"; block it and a page drops out of ChatGPT's cited answers, though it can still appear as a plain navigational link. OpenAI's own guidance names which token does which job: use OAI-SearchBot in robots.txt "for managing Search opt outs and automatic crawl," and GPTBot for training. A third agent, ChatGPT-User, fires when someone asks ChatGPT a question and the app fetches a specific page to answer it on the spot. That is a live, user-triggered request rather than a standing crawl, and OpenAI says plainly that "robots.txt rules may not apply" to it. Blocking GPTBot does not touch this third path either way.
Cloudflare's new setting fixes what "Block" used to skip entirely
Cloudflare's own words on the old behavior: "Block and 'Block on pages with ads' previously did not apply to mixed-use crawlers because blocking them could also affect search discoverability." In practice that meant Googlebot, Applebot and Bingbot kept crawling for both search and training, unrestricted, whether or not a site had Block turned on; a site owner had no lever inside Cloudflare to refuse just the training half of that job. The new "Disallow AI Training" setting is that lever. Cloudflare states the result plainly: "Accountable mixed-use crawlers remain allowed for search. Every other training crawler is blocked." The "Accountable" label carries the weight in that sentence. It only applies to an operator that meets four conditions Cloudflare states outright: a mechanism to opt out of training through robots.txt or an equivalent standard, a mechanism to opt out of AI summaries, page-level visibility into how content was used for training and how it performed in search, and an assurance that opting out of training carries no penalty in search results. Apple, Google and Microsoft's mixed-use crawlers meet that bar today; Amazon, Anthropic, Meta and OpenAI's training-only crawlers meet a version of it separately. Cloudflare reports that 17% of the sites on its network already enable some form of AI-training block and fewer than 1% block search outright. That figure is Cloudflare's own reading of its own network, not an independent survey, and worth holding to that scope.
The lines to add if a site is not behind Cloudflare
A site that manages its own robots.txt gets the same outcome for the two most cited AI crawlers without waiting on any vendor's dashboard:
``` User-agent: GPTBot Disallow: /
User-agent: Google-Extended Disallow: / ```
Leave Googlebot, Bingbot, OAI-SearchBot and ChatGPT-User out of the file, or set each to Allow: / if the rest of the file is written defensively. Check first for an existing block naming either GPTBot or Google-Extended already in the file, and edit that one rather than appending a second: the standard says two groups for the same agent get combined before a crawler reads them, but a rule about one crawler stated in two places is harder to audit the next time someone touches the file. OAI-SearchBot is worth adding as an explicit Allow rather than leaving it implicit, since it is the one line standing between a site and disappearing from ChatGPT's own search answers if the rest of the file changes later.
Go Voltic's scans already flag a robots.txt file that blocks AI crawlers as a single group, the blanket version of this same problem, because a rule written to stop one crawler often catches several a site never meant to turn away. The setting above narrows that decision to the specific token doing the training, instead of an all-or-nothing rule nobody checked twice.
Sources
- Have it both ways: stay discoverable in search while disallowing AI training. Cloudflare, read 2026-09-15
- List of Google's common crawlers. Google Search Central, read 2026-09-15
- GPTBot. OpenAI, read 2026-09-15
Run a free Instant Scan on your own site · More from the Newsroom