Who gets to use your content? AI crawler controls in 2026: robots.txt, Cloudflare and Google's AI opt-out
You can keep your content out of AI training and still appear in AI search, but only if you treat it as four separate decisions, not one “block AI” switch. As of October 2026 the levers are: robots.txt rules per bot (training, search and user-triggered fetchers are different bots), your CDN’s bot settings, Google-Extended for Gemini training, and a new Search Console control that takes your site out of AI Overviews and AI Mode without touching classic rankings. The one mistake to avoid: since 15 September 2026, the “Block” setting for AI training in Cloudflare also blocks Googlebot, Bingbot and Applebot, which means it blocks your search traffic too.
Most guides on this topic are lists of user-agent strings with a robots.txt snippet underneath. Several I looked at for this article still describe Cloudflare’s plans from July as if they had shipped unchanged, list bots the vendors themselves don’t document, or don’t mention that some AI fetchers don’t read robots.txt at all. Below is what the operators themselves document, as of October 2026, with links.
The short answer: block training if you want, never block search by accident
Three kinds of bots visit your site on behalf of AI products:
- Training crawlers collect content for future models. Blocking them has no documented effect on whether you appear in AI answers.
- Search crawlers build the index that AI assistants search before they answer. Block them and you drop out of those answers.
- User-triggered fetchers load a page because a person asked an assistant about it right now. Some vendors say these don’t follow robots.txt.
Blocking training is a legitimate business decision. Blocking AI search crawlers is usually an accident: a CDN default, a copied robots.txt, a security plugin. The rest of this article is about making the first decision on purpose and avoiding the second.
The bots that matter, and what each one does
This table only lists what each operator documents on its own crawler page.
| Operator | robots.txt token | What it’s for | What blocking it does |
|---|---|---|---|
| OpenAI | GPTBot |
Training | “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models” (OpenAI) |
| OpenAI | OAI-SearchBot |
ChatGPT search | Sites opted out “will not be shown in ChatGPT search answers, though can still appear as navigational links” |
| OpenAI | ChatGPT-User |
Fetches pages for user requests and GPT Actions | OpenAI: “Because these actions are initiated by a user, robots.txt rules may not apply.” It is not used to decide search inclusion |
| Anthropic | ClaudeBot |
Training | Signals that future content should be excluded from training (Anthropic) |
| Anthropic | Claude-SearchBot |
Search index | Prevents indexing for search, “which may reduce your site’s visibility” in Claude’s search results |
| Anthropic | Claude-User |
Fetches pages for user questions | Prevents retrieval in response to a user query |
| Perplexity | PerplexityBot |
Perplexity search; “not used to crawl content for AI foundation models” (Perplexity) | Removes you from Perplexity search results |
| Perplexity | Perplexity-User |
Fetches pages for user questions | Perplexity: “this fetcher generally ignores robots.txt rules” |
Googlebot |
Google Search, including AI Overviews and AI Mode | Removes you from Google Search | |
Google-Extended |
Gemini training and grounding in Gemini Apps and Vertex AI (Google) | “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” | |
| Apple | Applebot-Extended |
Training of Apple’s foundation models (Apple) | Content stays discoverable in Siri, Spotlight and Safari |
Some details that matter in practice:
- Anthropic documents three bots. Some guides list more Anthropic user agents; the Anthropic help page (version of 7 April 2026) lists ClaudeBot, Claude-User and Claude-SearchBot. It says all of them honor robots.txt, supports the non-standard
Crawl-delay, and warns that blocking its IP addresses instead “may not work correctly or persistently guarantee an opt-out”, because it stops the bot from reading robots.txt. - Google-Extended is a robots.txt token only. It has no user agent of its own; Google crawls with its normal crawlers and reads the token as a usage rule. You won’t find it in your logs.
- Changes are not instant. OpenAI says it takes about 24 hours for its systems to reflect a robots.txt change. Perplexity says up to 24 hours. Google caches robots.txt for up to 24 hours, longer if it can’t refresh the file.
- OpenAI keeps the two decisions separate. Its documentation says you can allow OAI-SearchBot and disallow GPTBot. If both are allowed, OpenAI may use one crawl for both purposes.
A robots.txt that refuses training and keeps AI search open looks like this:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /cart/Two rules from Google’s robots.txt specification catch people out here. Several User-agent lines can share one group, as above. And a crawler follows only the most specific group that matches its name; “other groups are ignored”. So once GPTBot has its own group, your User-agent: * rules no longer apply to it. If /cart/ should stay closed to everyone, it has to be in every group.
Your CDN may decide for you: Cloudflare’s 2026 changes
If your site runs behind Cloudflare, its bot settings decide what reaches your server before robots.txt is even read. Its plans changed during 2026, and articles written in July are now partly wrong.
1 July 2026. Cloudflare introduced three AI categories that every plan, including Free, can control separately: Search, Agent (fetchers acting for a person, such as ChatGPT-User) and Training. It announced that from 15 September, new domains would block Training and Agent on pages that show ads, and that mixed-use crawlers such as Googlebot would be blocked wherever Training was blocked.
21 August 2026. Bot Preference Sync arrived: Cloudflare writes your AI settings into your robots.txt, prepended to whatever is already there. It is on by default for new customers.
15 September 2026. What actually shipped differs from the July announcement:
- A new Training option, Disallow AI Training, publishes a no-training rule in robots.txt. Crawlers that Cloudflare classifies as “Accountable”, including Googlebot, Bingbot and Applebot, can still crawl for search. Every other training crawler is blocked.
- Block and Block on pages with ads now apply to mixed-use crawlers. Cloudflare’s own words: selecting Block “will stop Applebot, Bingbot, and Googlebot from reaching your site — search included.”
- The one-click “Block AI Bots” switch and Managed robots.txt are deprecated. Existing settings were migrated: domains that had Training set to Block were moved to Disallow AI Training, so they don’t suddenly lose Googlebot.
- New domains get one of two presets. Sites without ads: everything allowed. Sites that monetize with ads: Search allowed, Training set to Disallow AI Training, Agent blocked on pages with ads.
Two consequences for SEO. First, anyone who sets Training to “Block” after 15 September, because it sounds like the stronger option, takes the site out of Google and Bing. Second, Disallow AI Training does not yet reach Microsoft: Cloudflare writes that Bing’s support for a robots.txt no-training rule is targeted for early 2027 and that until then Bing’s NOARCHIVE tag is the way to opt out of training there.
There is a quieter effect too. If Bot Preference Sync is on, the robots.txt served to bots is no longer the file in your repository. Your developers may be editing a file that Cloudflare prepends rules to.
Google is a special case: four controls, four different jobs
Google uses one crawler, Googlebot, for Search and for its AI features in Search. That is why “block Google’s AI” questions are confusing. Google’s AI features documentation says “robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search”, AI included. Here is what each control does:
| Control | What it affects | What it doesn’t affect |
|---|---|---|
Disallow for Googlebot |
All of Google Search, AI features included | – |
Google-Extended |
Gemini model training, grounding in Gemini Apps and Vertex AI | Search inclusion and ranking |
nosnippet, data-nosnippet, max-snippet |
How much of a page Google may show as a preview in Search, AI features included | Indexing |
| Search Console: Settings > Search generative AI | AI Overviews, AI Mode, generative AI features in Discover | Classic results; “this control isn’t used as a ranking or inclusion signal affecting other parts of Search” |
The Search Console control is new. Google announced it on 3 June 2026 as a UK test; since 31 August 2026 it is available to all sites worldwide. The help page gives the details:
- Include is the default. Exclude means links to your site and your content won’t appear in those features, and your content won’t be used to ground AI responses there. “You won’t receive any traffic or impressions from these features.”
- Exclusion usually takes effect within a few days; some content takes longer because of caching.
- It doesn’t affect training. For that, the page points to Google-Extended.
- It isn’t only site-wide. Child properties (a subdomain or a URL-prefix property like
https://example.com/blog/) inherit the parent’s setting but can override it. So you could exclude one section and keep the rest in.
Before you exclude anything, look at the generative AI performance report in Search Console. Since 31 August it shows impressions in AI Overviews and AI Mode by page, country and device. It shows impressions, not clicks, so you need your analytics data alongside it to see what those features actually send you.
Decision matrix: which setting fits which business model
This is my starting point, not a rule. It assumes the business wants visitors, not licensing deals.
| Business model | Training bots | AI search bots | User-triggered fetchers | Google AI features (Search Console) |
|---|---|---|---|---|
| B2B lead generation, services | Allow or disallow, low stakes | Allow | Allow | Include |
| Ecommerce | Allow, unless product texts are a real asset | Allow | Allow | Include |
| Ad-funded publisher | Disallow | Allow | Your call: these visits don’t see ads | Measure first, then decide per section |
| Affiliate | Disallow if the content is original research | Allow | Allow | Measure first |
The reasoning behind it:
- B2B and services. A buyer who asks an assistant for a shortlist of providers should find you. Whether your text is in a training set matters little next to that. Training is an opinion call; search access is not.
- Ecommerce. Product pages are meant to be found and copied into comparisons. Blocking training rarely protects much, unless your descriptions, guides or reviews are what sets you apart.
- Publishers. Training takes the content without a visit; agents fetch it without seeing ads. That is the logic behind Cloudflare’s ad-site preset. The Search Console exclusion is the expensive lever: it also removes the links in AI Overviews. Decide it with data from the performance report, and by section if your sections differ.
- Affiliates. The value sits in the click to the partner. If AI answers quote your comparison without a visit, that hurts; if they link you, it helps. That depends on the query type, so measure before excluding.
Google states for both Google-Extended and the Search Console control that they are not used as ranking signals for classic results.
How to check what actually happens
robots.txt is a statement of intent. What bots get is decided by your server, CDN and security plugins. In an audit I check both, in this order:
- Fetch the live robots.txt, not the one in your repository:
curl -s https://example.com/robots.txt. Look for aBEGIN Cloudflareblock or other rules you didn’t write. - Check groups and precedence. Does every AI bot group repeat the rules it must keep (as in the
/cart/example above)? Does anything accidentally matchGooglebotorOAI-SearchBot? To check single URLs, test your robots.txt. - Request the page as each bot. My free AI Crawler Check tests 13 AI bots against your robots.txt and requests the page with each bot’s user agent. If a browser gets a 200 and a bot gets a 403 or 429, a firewall rule is blocking it, whatever robots.txt says. One limit: a test like this sends the user agent from a normal IP. Rules that verify the bot’s IP address only show up in your logs.
- Read your server or CDN logs. Filter by user agent and status code. Verify that a request really comes from the operator: Google documents a reverse DNS check; OpenAI, Anthropic and Perplexity publish IP lists linked from the documentation pages above.
- Check the CDN settings. In Cloudflare, look at the Training, Search and Agent settings at domain level and confirm that Training is not on “Block” unless you mean to leave Google.
- Check Search Console. The robots.txt report shows which robots.txt files Google found, when it last crawled them and any errors. Settings > Search generative AI shows whether a property, or a parent property above it, excludes AI features.
If you want to go beyond access and look at what drives citations, my AI search checklist goes through crawler access, rendering, quotable content, entities and measurement, and the FAQ rich results article covers why Google says you don’t need special files or markup for its AI features.
What I’d do this week
Write down one sentence per bot category: training yes or no, AI search yes, user fetchers yes or no, Google AI features include or exclude. Then make robots.txt, the CDN and Search Console say the same thing, and check the result from the outside. The usual failure is not a wrong decision but a setting nobody decided: a plugin default, a migrated CDN setting, a robots.txt rule copied from a blog.
If you’d like that mapped for your site, including what AI search currently sends you, that’s part of my AI search work.