Pagination and Faceted Navigation

Category pages split across many pages, and filters that multiply into thousands of near-identical URLs, are where large sites bleed crawl budget. The rules for the two are different, so this chapter keeps them separate: how to paginate a list without hiding your products, and how to stop faceted navigation from drowning the index.

Pagination done right

Start by dropping an old habit. rel="next" and rel="prev" are no longer used by Google, confirmed in 2019. Keep them if you like for Bing, accessibility, and browser prefetch, but do not rely on them to tie your pages together for Google.

The rules that do matter for a paginated series:

• Give each page a unique, crawlable URL like /category?page=2, not a fragment or a JavaScript-only state Google cannot request.
• Let each page self-canonicalize. Page 2 canonicals to page 2, not back to page 1. Canonicalizing every page to the first hides the deeper items from crawling, because Google follows the canonical and never indexes page 2's contents.
• Link the pages together and back to the first, so Google can walk the series and reach the deep items.
• Keep deep items reachable, and avoid noindex on paginated pages if that series is Google's only path to your detail pages.

The through-line: pagination exists to make deep content reachable, so every rule protects reachability. Break it and your newest or lowest products quietly fall out of the index.

The facet problem

Faceted navigation is the filter and sort controls on a listing: color, size, price, brand, sort order. The trouble is that each combination often generates a new URL, and combinations multiply. A handful of filters can spawn thousands or millions of near-duplicate URLs. On large and ecommerce sites this is the single biggest crawl-budget drain there is, and fixing it is the highest-return technical job on the site. Facets as the top crawl drain →

The damage is double. Google wastes crawl capacity fetching endless filter permutations instead of your real pages, and the index bloats with thin, duplicative combinations that dilute how Google sees the site. Left alone, facets can consume most of your crawl budget while adding nothing worth indexing.

The combinatorial math is what makes it dangerous. Five filters with a handful of options each do not add a few URLs, they multiply into thousands, and adding sort orders and result-per-page options multiplies again. Session IDs and tracking parameters can push it further still. This is why faceted navigation, not pagination, is the crawl problem that actually sinks large sites, and why Google singles it out as the classic crawl-budget drain.

Decide what to index first

Before touching any tags, make one decision: do you want any facet URLs indexed at all? For many sites the answer is no, and the whole facet space should be kept out of search. For others, a few high-demand filters (a popular brand-plus-category page people actually search for) are worth indexing as landing pages. That decision drives everything after it, and skipping it is why so many sites end up with a half-controlled mess. Decide the outcome, then apply the mechanism.

Decide whether you want facet URLs indexed before you configure anything. The choice, all out or a curated few, determines which controls you use, and applying controls without that decision creates the mess you were trying to avoid.

The controls

For facets you do not want indexed, block them from crawling by disallowing the patterns in robots.txt. That saves the most crawl budget, because Google never fetches the URLs at all. Where the URLs stay reachable and you want a softer touch, nofollow the facet links and apply a canonical or noindex to the combinations. Blocking the crawl is the stronger crawl-budget move, and it lives in the robots chapter. robots.txt to block facet patterns →

For the few facets you do want indexed, keep them clean. Use the standard & separator for parameters, not commas, semicolons, or brackets. Keep the filter order stable so one selection does not generate several URLs for the same result. Return a real 404 for empty combinations rather than an empty 200 that reads as a soft 404. And canonicalize non-preferred combinations to the preferred one so their signals consolidate rather than compete. Canonicalize facet variants →

You cannot control what you have not measured, so inventory the actual patterns first, pulling them from server logs and a site crawl, then classify each as index-worthy or waste before you apply anything. Whatever you keep out of the index, keep out of your sitemaps too, so you are not asking Google to index the very URLs you are trying to suppress. Reading those logs is the discipline the next chapter covers. Log file analysis next →