The Engines and Their Crawlers

Every answer engine sends its own crawlers, and each crawler does one of three jobs: train a model, index for search, or fetch a page live when a user asks. Confuse the three and you will block the wrong bot, cut yourself out of answers, and think you opted out of training when you did not. This chapter maps who is who.

How do you show up in Google AI Overviews?

You do nothing special. Google states plainly that there are no additional requirements to appear in AI Overviews or AI Mode, and no new machine-readable file or schema to create. If you meet standard Search requirements, are crawlable and indexable, and keep your substance in text, you are eligible. AI Overviews sit above or inside results as a summary block; AI Mode is the conversational tab. Both draw from the same index that feeds normal search.

Two traps follow from that. First, you cannot opt out of AI Overviews on their own while staying in Search. The only levers are nosnippet, max-snippet, data-nosnippet, and noindex, and each of those also suppresses your normal search snippet, so the cure is worse than the disease for most sites. Second, blocking Google-Extended does not remove you from AI Overviews. That token is training-only. It controls whether your content trains future Gemini models, and nothing more. People block it expecting to disappear from AI answers and are surprised when they do not. Snippet controls →

What do ChatGPT and Perplexity crawl with?

ChatGPT with search browses the live web and cites sources. To surface there, be crawlable by OAI-SearchBot, which is OpenAI's search crawler. A common mistake is to block GPTBot and assume you are out of ChatGPT. You are not. GPTBot only affects training data. If you want to appear in ChatGPT answers, leave OAI-SearchBot allowed. There is also ChatGPT-User, which fetches a page live when a user asks for it in the moment.

Perplexity is answer-first, with prominent numbered citations next to each claim. It runs PerplexityBot for indexing and Perplexity-User for live user fetches. Its robots.txt compliance is disputed, and you should present both sides. In August 2025 Cloudflare alleged that Perplexity used stealth crawlers to bypass robots.txt on sites that had blocked it. Perplexity's position is that a fetch triggered by a specific user is not a crawler and is not bound by robots.txt the way a bulk indexer is. Treat it as an open controversy, not a settled fact, and assume robots.txt alone may not fully stop a user-triggered fetch.

What about Claude, Gemini, and Copilot?

Claude uses three tokens. ClaudeBot gathers training data, Claude-User fetches a page when a user asks, and Claude-SearchBot indexes for Claude's search. Older tokens like Claude-Web and anthropic-ai still turn up in logs. Anthropic states that it honors robots.txt and Crawl-delay, so the standard controls apply.

Gemini grounds its answers through Google Search, so the same crawlable-and-indexable rules that win you a place in Search apply here too. It will also train on your content unless you disallow Google-Extended. Microsoft Copilot builds on the Bing index rather than a crawler of its own, so the requirement is simple: be indexable in Bing. That makes Bing Webmaster Tools and IndexNow more useful than they look, because Bing indexing now feeds AI answers, not just Bing's own search share. Bing and IndexNow →

The single most common error is blocking a training bot and believing you left the answers. Training, search, and live fetch are three different jobs done by three different tokens. Block the wrong one and you vanish from citations while your content still trains models.

Which token does what?

Here is the quick reference. Training tokens build future models. Search tokens index you so you can be cited in answers. Live-fetch tokens pull a single page when a user asks for it right then.

• OpenAI: GPTBot trains, OAI-SearchBot searches, ChatGPT-User fetches live.
• Anthropic: ClaudeBot trains, Claude-SearchBot searches, Claude-User fetches live.
• Perplexity: PerplexityBot indexes, Perplexity-User fetches live (compliance disputed).
• Google: Googlebot indexes for Search and feeds AI Overviews; Google-Extended is a training-only control, not a fetching bot.
• Microsoft: bingbot indexes for Bing, which Copilot reads.

Re-check these against each operator's own documentation before you act on them, because user-agent versions and token names change, and vendors add or retire bots without much notice. The tokens matter most when you decide what to allow and what to block, which is a policy you set in robots.txt. robots.txt → Once the engines can read you, the next move is earning the citation itself with how you write the page. Getting cited →