Sitemaps and URL Discovery

A sitemap is a list of the URLs you want engines to know about. It helps Google discover pages, especially on large, new, or poorly linked sites. What it is not is a promise those pages get indexed. Treat it as a discovery hint you keep clean, not a lever that forces anything into the index.

What a sitemap is

An XML sitemap lists URLs, each optionally paired with a lastmod date noting when the page last changed. Its job is discovery: it tells Google "these URLs exist and here is when they moved," which speeds finding pages that internal links alone might reach slowly or not at all. On a small, well-linked site Google will find your pages anyway. The value grows with size and with how poorly a page is linked internally.

Set expectations correctly. A sitemap is a hint, not a guarantee of indexing. Listing a URL asks Google to consider it; Google still decides whether the page is worth keeping. Discovery is upstream of the crawl budget question, because a URL Google has not discovered cannot be crawled at all. Crawl budget →

A sitemap does not replace internal linking; it complements it. Google discovers most pages by following links, and a page that is well linked from the rest of your site will usually be found with or without a sitemap. The sitemap earns its value in the gaps: a brand-new site with few inbound links, a very large site where some pages sit many clicks deep, or a section that is poorly connected internally. If your important pages depend on the sitemap to be found at all, that is a sign your internal linking needs work, not just a bigger sitemap.

The limits

The format has hard ceilings. A single sitemap file holds at most 50,000 URLs and 50MB uncompressed. When you exceed either, split the URLs across multiple files and reference them from a sitemap index, which itself caps at 50,000 sitemaps and 50MB. Gzip compression is supported, and the size limit is measured on the uncompressed file, so compression saves bandwidth but not the 50MB budget.

The rule that keeps a sitemap useful is what you allow into it. Include only canonical URLs that return 200, are indexable, absolute, and on the same site. Listing redirects, noindexed pages, 404s, or non-canonical variants sends Google mixed signals and wastes its attention. A sitemap should be a clean statement of the pages you actually want indexed, which means every URL in it should be one you would declare canonical. Only canonical URLs belong →

lastmod, priority, and changefreq

lastmod is the one optional field that earns its place, but only if it is honest. An accurate lastmod is a recrawl hint: it tells Google a page genuinely changed and is worth a fresh look. Fake it, or stamp every URL with the same date on each build, and Google learns the field is noise and ignores it across your whole sitemap. Accuracy is what makes it worth including at all.

The other two legacy fields are effectively dead. Google largely ignores priority and changefreq, so do not spend effort tuning them. And the old sitemap ping endpoint, where you pinged a URL to announce an update, was deprecated in 2023. The current way to make Google aware of your sitemaps is to list them in robots.txt with a Sitemap: line and submit them in Search Console.

An accurate lastmod is the only sitemap metadata Google still trusts. Priority and changefreq are ignored, and the ping endpoint is gone, so list sitemaps in robots.txt and Search Console instead.

Build and monitor

The workflow is short. Generate a sitemap containing only canonical, indexable, 200 URLs. Split it into files under the 50,000-URL and 50MB limits, tied together by a sitemap index if needed. Set accurate lastmod values. Validate the file against the sitemaps.org schema so a malformed entry does not sink the whole file. Reference it in robots.txt, and submit it in the Search Console Sitemaps report.

Then monitor. In the Page Indexing report, watch the "Submitted URL not indexed" bucket, the pages you asked Google to index but it declined. A growing count there is Google telling you those pages are not clearing the quality or value bar, which points back at content, not at the sitemap. Treat the sitemap as a living file, not a one-time upload: regenerate it when you publish, remove, or redirect pages, so it always reflects the canonical set you actually want indexed rather than a stale snapshot.

For faster discovery on engines that support it, IndexNow lets you push changed URLs directly to Bing and some AI-adjacent crawlers rather than waiting for a recrawl. IndexNow for Bing and AI → Sitemaps handle discovery at the whole-site scale; the next challenge is controlling the flood of URLs that pagination and filters generate. Pagination and facets next →