Robots and Crawl Control
Two different controls get confused constantly, and the confusion breaks pages. One decides whether Google may fetch a URL. The other decides whether Google may index or show it. They live in different places, do different jobs, and combining them on one URL cancels the effect you wanted. Get this distinction right and most crawl-control mistakes disappear.
Two jobs, never combined
robots.txt controls crawling: which URLs a compliant bot may request. The robots meta tag and the X-Robots-Tag header control indexing and serving: whether a fetched page may be stored and shown. The trap is treating them as interchangeable. They are not. A crawl block and an index directive on the same URL fight each other, because Google has to fetch a page to read any indexing directive on it. Block the fetch and the directive is never seen.
So the rule is one job per URL. If you want a page gone from the index, let Google crawl it and addnoindex. If you want to stop Google fetching a URL, block it in robots.txt and accept that any noindex on that page becomes invisible. Deciding which control you actually need is the first move, every time.
robots.txt controls crawling
robots.txt is a plain-text file at the domain root telling compliant crawlers which paths they may request. It follows the Robots Exclusion Protocol, standardized as RFC 9309 in 2022. Think of it as a doormat note, not a lock: well-behaved bots obey it, bad actors ignore it. The directives are User-agent, Disallow, Allow, and Sitemap, and Google supports * as a wildcard and $ to anchor the end of a URL.
The facts that trip people up:
- It controls crawling, not indexing. A URL you block in robots.txt can still be indexed, URL only with no snippet, if other pages link to it. Blocking is not a way to keep something out of search.
noindexinside robots.txt is unsupported. Google dropped it on 1 September 2019. To keep a page out of the index, allow crawling and use anoindexmeta tag or header instead.- Do not block the CSS and JavaScript needed to render the page. Block those and Google renders a broken version.
- Keep the file reachable and returning
200. A long stretch of 5xx responses on robots.txt can make Google treat the whole site as disallowed, though the exact fallback timing has shifted and is not firmly published.
Blocking a fetch is the only crawl control that truly saves crawl budget, which is why the crawl chapter leans on robots.txt. Crawl budget →
Meta robots controls indexing
The robots meta tag lives in the <head> as <meta name="robots" content="noindex">, and the X-Robots-Tag HTTP header does the same job for responses. Common values include noindex (keep out of the index), nofollow (do not follow links from this page), none (both), nosnippet, max-snippet:[n], and unavailable_after:[date]. Use the meta tag for HTML pages. Use X-Robots-Tag for non-HTML files like PDFs and images, or for server-wide rules where editing every page is impractical.
The dependency to memorize: for noindex to work, the page must be crawlable. Google can only obey a directive it has fetched and read. This is the mechanism behind index management, where noindex is the tool for keeping thin pages out of the index while still letting Google reach them. Index management →
robots.txt decides what Google may fetch. The robots meta tag and header decide what Google may index. Put both on one URL and the block hides the directive, so neither does what you meant.
The traps
Almost every crawl-control failure is one of a few repeated mistakes. First, blocking a page in robots.txt to keep it out of the index. The block hides the noindex Google would otherwise read, so the URL can linger in results as a bare link. Second, combining a robots.txt block and a noindex on the same page and expecting both to fire. Only one can, and the block wins, so the page stays indexable-in-name.
Third, a rendering trap. If your raw HTML ships a noindex and your JavaScript later swaps it to index, Google may act on the initial noindex before or instead of the change. A JavaScript-added index cannot reliably override a raw-HTML noindex. Set the directive you actually want in the initial HTML and confirm it survives rendering.
One decision covers more than classic search. AI answer engines run their own crawlers, and their access tokens ride the same robots.txt model you just learned. Set crawler policy once and it governs both Google and the AI fetchers. Controlling AI crawlers → With crawling and indexing settled, the next question is which URL Google should treat as the real one among duplicates. Canonicalization next →