Crawling Foundations
Everything in technical SEO rests on one floor: a page Googlebot never fetches can never be indexed or ranked. Crawling is that floor. Get it wrong and your best content is invisible; get it right and you stop wasting Google's attention on pages that do not matter.
What crawling is
Crawling is a bot requesting URLs over HTTP, downloading the responses, and following the links it finds to discover more URLs. Google's bot is Googlebot. It starts from URLs it already knows, plus sitemaps and links, fetches each one, and queues the new links it sees for later. The pages it renders and stores come from this loop. Break the loop, by blocking a fetch or by never linking to a page, and that page drops out of everything downstream.
Crawl budget
Crawl budget is the number of URLs Googlebot can and wants to crawl on your site in a given period. It is the product of two things. Crawl capacity is how much crawling your server can take without slowing down; Google backs off when response times rise or errors climb, and crawls more when the site stays fast and healthy. Crawl demand is how much Google wants to crawl, driven by how popular and how fresh your pages are. Google crawls only as fast as your server stays healthy, and only as much as it thinks your content is worth. Server speed and crawl capacity are directly linked. Server speed and TTFB →
When it actually matters
For most sites, crawl budget is not a real problem, and worrying about it is a distraction. Google itself says it matters mainly for large sites (roughly a million or more pages that change weekly, or tens of thousands that change daily) and for sites that spew low-value URLs. The classic culprit is faceted navigation, where filters and sort options multiply into millions of near-duplicate URLs.
The levers Google names are direct: raise server capacity, raise content value, keep sitemaps current, remove long redirect chains and soft 404s, and stop Googlebot from crawling URLs that do not deserve it. That last one is the single biggest win on large sites, and it depends on the crawl controls in the robots chapter. Robots and crawl control → Filter URLs are the usual drain. Pagination and facets →
The crawl-budget myth to kill
Here is the mistake almost everyone makes. Adding noindex, nofollow, or a canonical tag does not save crawl budget. Google has to crawl the page to see any of those tags in the first place. The only ways to save crawl are to block the fetch in robots.txt or to stop linking to the URL so Google never queues it.
noindex does not save crawl budget. Google must fetch the page to read the tag. Only blocking the fetch, or not linking the URL, actually saves crawl.
A related trap: if you block a URL in robots.txt, Google never fetches it, so it never sees a noindex on that page either. Choose one job per URL. Block crawling, or allow crawling and set noindex, never both on the same page expecting both to work.
Verifying Googlebot
Not every visitor claiming to be Googlebot is Googlebot. The user-agent string is trivial to fake, and scrapers do it constantly. To confirm a real Googlebot hit, do a reverse DNS lookup on the IP, check that it resolves to a googlebot.com or google.com host, then do a forward lookup to confirm it points back to the same IP. Or match the IP against Google's published ranges. Never trust the user-agent alone. This matters the moment you read server logs, which are the only ground truth for how bots actually crawl you. Log file analysis →
One more forward link worth planting now: AI answer engines run their own crawlers, and they reuse the same robots model you are about to learn. Decide crawler policy once and it covers both classic search and AI visibility. Controlling AI crawlers →