Controlling AI Crawlers

You can decide which AI crawlers read your site, but the controls are softer than most people assume. robots.txt is a request, not a lock, and the only hard enforcement lives at the edge. This chapter covers what each control actually does so you block the right bots and verify the block worked.

What can robots.txt actually control?

Less than its reputation suggests. robots.txt is voluntary. It tells compliant crawlers which paths they may request, and well-behaved bots honor it, but nothing forces them to. It controls crawling, not indexing: a URL you disallow can still appear as a bare link in results if other pages point to it. And it does nothing about what a model already absorbed. Blocking a training bot today does not remove your content from a model that trained on it last year. The full mechanics of the file live in its own chapter. robots.txt →

So set expectations before you set rules. robots.txt is the front-door sign that stops honest visitors and future crawls. It is not a way to claw back content already ingested, and it is not a guarantee against a crawler that decides to ignore it. The protocol was standardized as RFC 9309 in 2022, which formalizes the directives but not enforcement: compliance stays a choice each operator makes. Keep the file reliably reachable and returning a 200 status, because a long stretch of server errors can make some engines treat the whole site as disallowed, which is the opposite of what you intended.

Which token blocks which behavior?

To block deliberately you have to know what each token does, per operator. Training bots build future models. Search bots index you so you stay citable in answers. Live-fetch bots pull a single page when a user asks for it.

• OpenAI: GPTBot trains, OAI-SearchBot indexes for ChatGPT search, ChatGPT-User fetches live.
• Anthropic: ClaudeBot trains, Claude-User fetches live, Claude-SearchBot indexes for search.
• Perplexity: PerplexityBot indexes, Perplexity-User fetches live.
• Common Crawl: CCBot builds an open-web corpus that many models train on, so blocking it matters because third parties reuse that corpus.
• Google: Google-Extended is training-only and is not a fetching user-agent, so it never removes you from Search or AI Overviews.

How do you block training but stay citable?

The move most sites want is to opt out of model training while staying in AI answers. You do that by disallowing the training and corpus tokens and leaving the search tokens allowed. A starter policy blocks these in robots.txt:

• GPTBot Disallow: /
• ClaudeBot Disallow: /
• CCBot Disallow: /
• Google-Extended Disallow: /
• anthropic-ai Disallow: / (legacy token still seen in logs)

Leaving OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot allowed keeps you retrievable and citable in AI answers while opting out of training. That is the balance most publishers actually want: keep the visibility, decline to be training fuel. Blocking CCBot is worth its own line, because the Common Crawl corpus is reused by many third parties, so one disallow there reaches further than a single vendor's training bot. Decide this policy deliberately rather than copying someone else's file wholesale. A content business protecting original research leans toward blocking training; a company that wants maximum reach may allow everything and accept the training use as the cost of visibility.

Blocking every AI token is not caution, it is self-harm. Cut the search bots and you delete yourself from the answers while the training bots you meant to stop may have already read you. Block training, keep search.

Why is the edge your only hard control?

Because a fetch triggered by a specific user may not be bound by robots.txt at all. Several operators treat a live user-initiated fetch as different from a bulk crawl, so a disallow line may not stop it. That leaves one place where you can actually enforce a rule: the edge, at your CDN or server, where you can refuse or challenge a request before it reaches your content. Edge enforcement →

Enforcement also depends on knowing who is really hitting you, and spoofing is common. Anyone can put GPTBot in a user-agent string, and plenty of scrapers do exactly that to look legitimate. So verify rather than trust the label. Pull your server and CDN logs, filter by the tokens above, and confirm authenticity by reverse-DNS and by matching the source IP against the operator's published address ranges, not by trusting the user-agent name. Log file analysis → After you deploy a block, check the logs again to confirm the compliant bots stopped hitting the disallowed paths once the change propagated. Once you can see and control who reads you, the next question is whether any of it is working, which means measuring your visibility in the answers themselves. Measuring AI visibility →