AI Crawlers: Should You Allow Them or Block Them?

Most advice about AI crawlers treats this as one decision — allow AI, or block it. It is not one decision. Each of the major companies runs separate bots for separate purposes, and the consequences of blocking them point in opposite directions.

Getting this wrong is common and expensive: the usual mistake is blocking everything with "AI" in the name to protect your content from training, and quietly removing yourself from the answers your customers are reading.

Key Takeaways

• OpenAI runs GPTBot for training and OAI-SearchBot for search. Blocking the second means "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers" (OpenAI).
• Anthropic splits the same way: ClaudeBot for training, Claude-SearchBot and Claude-User for retrieval (Anthropic).
• PerplexityBot "is not used to crawl content for AI foundation models" — it is the search crawler (Perplexity).
• User-initiated fetches often ignore robots.txt, by design and by the companies' own statement.
• A named group in robots.txt replaces the wildcard group for that agent rather than adding to it. This is where most real mistakes happen.

The three jobs, not one

Read the vendors' own documentation and the same structure appears at all three companies.

Training crawlers. OpenAI's GPTBot is "used to make our generative AI foundation models more useful and safe" and to "crawl content that may be used in training"; disallowing it "indicates a site's content should not be used in training generative AI" (OpenAI). Anthropic's ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training" (Anthropic).

Search crawlers. OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features", and OpenAI states plainly that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers". Anthropic's Claude-SearchBot "navigates the web to improve search result quality for users", and blocking it "may reduce your site's visibility and accuracy in user search results". PerplexityBot "is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models" (Perplexity).

User-initiated fetches. ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked a question that needed it. Anthropic notes that blocking Claude-User "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search".

So "block AI crawlers" resolves into at least three different intentions, and only one of them is about training.

The mistake that costs the most

It is this: blocking the search crawler while meaning to block the training crawler.

The names encourage it. GPTBot and OAI-SearchBot are both OpenAI, both look like "the ChatGPT bot", and a copied robots.txt snippet from a forum post rarely explains the difference. The result is a site that is still in the training data of models trained before the block, and is no longer eligible to be cited in the answers its customers actually read.

If you take one thing from this: the search crawlers are the ones that put you in front of people. In a world where users increasingly stop at the answer, being in it is the visibility.

robots.txt behaves in a way that surprises people

A named user-agent group in robots.txt replaces the wildcard group for that agent rather than adding to it — which is where most real, silent failures come from.

A named user-agent group in robots.txt replaces the wildcard group for that agent — it does not add to it. So a file like this:

User-agent: *
Disallow: /admin/
Disallow: /internal/

User-agent: GPTBot
Allow: /

does not keep GPTBot out of /admin/. GPTBot follows only its own group, and its own group says everything is allowed. Every disallow you care about has to be repeated inside every named group.

We hit exactly this on our own site, which is why our robots.txt repeats the same disallow list under every named agent rather than relying on the wildcard.

What we do, and why

We allow all of them, and the reasoning is specific rather than ideological.

Caliency sells services, not access to text. Our content exists to be found by someone deciding whether to hire an agency. Being quoted in an answer someone reads instead of a search results page is not a loss of value — it is the point. The cost of the alternative is straightforward: being absent from the surface where the decision now gets made.

That calculation flips if your business is selling the content itself — a publisher, a course, a paid research product. Then the training crawlers are taking the inventory, and blocking them is a defensible commercial decision that costs you nothing in search visibility, because the search crawlers are separate. That separation is the whole reason this is worth understanding.

While writing this we checked our own file against the current documentation and found it named Claude-Web and anthropic-ai — names that are not in Anthropic's current article — while missing Claude-User and Claude-SearchBot, which are. Nothing was broken, because our wildcard group allows everything. But a stale list on a site that sells GEO is a poor advertisement, so we updated it.

A workable default

Allow the search crawlers. OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User. These decide whether you can be cited. We measured how often 175 popular Turkish sites block these crawlers in how to get cited by ChatGPT.

Decide the training crawlers on principle. GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended. There is no search cost to blocking them and no search benefit to allowing them. Answer the question you are actually being asked: are you willing for this content to contribute to model training?

Do not confuse Google-Extended with Googlebot. Google-Extended governs generative AI use of your content. Googlebot governs whether you are in Google Search at all. Blocking the wrong one is the most expensive typo available in this file.

Repeat your disallows inside every named group. See above. This is the failure that leaves an admin path crawlable for exactly one bot.

Then check it. Fetch your own robots.txt and read it as the bot would, group by group. It is a five-minute job that nobody does.

If you want the whole picture rather than just the crawler layer, our guide to generative engine optimization covers what the research actually supports — including how little of the popular advice replicates.

In short

There is no single "AI crawler" decision. There is a training decision, a search decision, and a user-fetch decision that robots.txt may not control at all. The companies publish which is which; the confusion is entirely on our side of the fence.

If you would rather have someone read your robots.txt properly than guess at it, that is a short conversation.