AI Search · · 5 min read · By Mobilize Cloud
AI companies send several different crawlers to websites, and they don't all do the same job. Some collect pages to train AI models. Some build a search index so an assistant can find and cite you. Others fetch a single page because a person just asked a question about it. In robots.txt you can treat each one differently, which means you can opt out of AI training without disappearing from AI search.
Below is a list of the main AI crawlers and what their owners say they do, the tradeoffs of allowing or blocking them, and sample rules you can copy. Crawler names and policies change, so everything here is as of October 2026, with links to each company's own page so you can check.
A quick refresher on robots.txt
robots.txt is a plain text file at yourdomain.com/robots.txt. Each group of rules starts with a User-agent line naming a crawler, followed by Allow or Disallow lines for parts of your site. If a crawler isn't named, it follows the User-agent: * group, and if there's no rule at all, it's allowed.
Two things to keep in mind. robots.txt is a request, not a lock: reputable crawlers follow it, but it can't technically stop anyone. And it's a different file from llms.txt. robots.txt says who may crawl; llms.txt suggests what's worth reading. Our post on llms.txt vs robots.txt vs sitemap.xml explains how the three fit together.
The main AI crawlers in robots.txt
We've grouped them by what they do, because that's what matters when you decide.
Crawlers that collect content for AI training
| User agent | Owner | What the owner says it does |
|---|---|---|
GPTBot | OpenAI | Crawls content that may be used to train OpenAI's foundation models. |
ClaudeBot | Anthropic | Collects web content for training. Blocking it signals your future content should be excluded from training data. |
Google-Extended | Not a separate crawler but a control token. Decides whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it doesn't affect inclusion or ranking in Google Search. | |
Applebot-Extended | Apple | Also a control token. Lets you opt out of Apple using your content to train its foundation models. Apple says it isn't used for search ranking. |
meta-externalagent | Meta | Crawls for uses such as training AI models or improving products. |
CCBot | Common Crawl | Builds Common Crawl's open web archive, which is widely used to train AI models. |
Crawlers that power AI search and citations
| User agent | Owner | What the owner says it does |
|---|---|---|
OAI-SearchBot | OpenAI | Finds websites to show in ChatGPT's search features. Separate from GPTBot. |
Claude-SearchBot | Anthropic | Indexes content to improve Claude's search results. Anthropic says blocking it may reduce your visibility. |
PerplexityBot | Perplexity | Surfaces and links websites in Perplexity's results. Perplexity says it isn't used to train foundation models. |
Meta-WebIndexer | Meta | Improves search results in Meta AI. |
Googlebot | Google Search. Google's AI Overviews and AI Mode draw on pages that are indexed and eligible for Search. | |
Applebot | Apple | Powers search in Spotlight, Siri and Safari. Apple says its data may also be used for training unless you block Applebot-Extended. |
Bingbot | Microsoft | Bing search, whose index Microsoft's Copilot also draws on. |
Fetchers that act when a person asks
| User agent | Owner | Does it follow robots.txt? |
|---|---|---|
ChatGPT-User | OpenAI | OpenAI says robots.txt rules may not apply, because a user started the request. |
Claude-User | Anthropic | Anthropic says it respects robots.txt, and that blocking it may reduce your visibility. |
Perplexity-User | Perplexity | Perplexity says it generally ignores robots.txt, because a user requested the fetch. |
meta-externalfetcher | Meta | Meta says it may bypass robots.txt for user-requested fetches. |
Sources, worth bookmarking because they change: OpenAI's crawlers, Anthropic's crawlers, Perplexity's crawlers, Google's common crawlers, About Applebot, Meta's web crawlers and the Common Crawl FAQ.
Allow or block? The tradeoffs
There's no single right answer. It depends on what you publish and what you want from AI tools.
Blocking training crawlers is a reasonable choice, especially for publishers, creators and anyone whose writing is the product. According to OpenAI, Google and Apple's own documentation, blocking their training tokens doesn't remove you from their search features. The catch: it only applies going forward. It doesn't pull back anything already collected.
Blocking search crawlers is different. If OAI-SearchBot, Claude-SearchBot or PerplexityBot can't read your pages, those assistants have less to go on when someone asks about your business, and Anthropic and others say so plainly. For most local businesses, shops and service companies, the people asking AI tools for recommendations are potential customers, so blocking search crawlers works against you.
Blocking user-triggered fetchers often doesn't work reliably, since some say they may not follow robots.txt. They also only fetch a page because a person is asking about it, which is usually a good thing for you.
Don't block Googlebot to stay out of AI Overviews. That would remove you from Google Search entirely. Google points site owners to nosnippet, max-snippet or noindex controls instead (see AI features and your website).
Sample robots.txt rules
Pick the one that matches your choice. Crawler names aren't case sensitive, and you can list several User-agent lines above one set of rules.
Allow everything (the default)
If your robots.txt doesn't mention AI crawlers and has no Disallow: / for everyone, they're already allowed. You don't need to add anything.
Block AI training, keep AI search
# Opt out of AI training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot Disallow: / # Everyone else, including search crawlers User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
The second group is a typical WordPress example. Keep whatever your existing User-agent: * group already says.
Keep one folder away from all AI crawlers
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: PerplexityBot Disallow: /members/
Remember that anything truly private should sit behind a login. robots.txt is public, so it also tells people which folders you'd rather they didn't look at.
Block all the AI crawlers listed above
You can list every token from the tables above with Disallow: /. Do it only if you've weighed the cost: AI search tools will have much less to say about you, and the user-triggered fetchers may still visit.
Check you're not blocking AI crawlers by accident
The most common problem we see isn't a deliberate choice. It's a block nobody meant to add: a security plugin, a CDN or firewall setting, a website builder option to block AI crawlers, or an old Disallow: / left over from when the site was in development. Some CDNs, Cloudflare among them, offer settings that ask AI crawlers to stay away or block them outright, so check your dashboard as well as the file itself.
Our free AI readiness check reads your robots.txt and shows, crawler by crawler, which AI search tools can reach your site and which training crawlers you block, then explains what each choice means. It treats blocking training crawlers as your call, not a fault. While you're there, run your llms.txt through the validator too; how to check if your llms.txt is working walks through the results.
Bottom line: AI crawlers come in three kinds: training, search and user-triggered. If you want to opt out of training, block the training tokens and leave the search crawlers alone. Check the official pages before you edit, because the list keeps changing.
Is your llms.txt working?
Check any site for free and get plain-English fixes in seconds.