Indexing and Index Management

Crawled is not indexed. A page can return a clean response, sit in Google's systems, and still never appear in search. Indexing is the second gate, and it is where quality gets judged. Learn what makes a page indexable and how to read the report that tells you why one is not.

What indexing is

Indexing is storing and organizing a crawled, rendered page so Google can return it for a query. The key point people miss: crawling and indexing are separate steps. Googlebot fetching a URL does not mean Google keeps it. A 200 status does not guarantee inclusion either. Thin, duplicate, or low-value pages get crawled, looked at, and dropped. The index is curated, not a mirror of everything Google can reach.

This is why "my page returns fine in the browser but it is not in Google" is such a common complaint. The page is reachable. It just did not earn a slot. Ranking is a problem you only get to have after a page is indexed, so index status is the first thing to check when a page is missing from search.

What makes a page indexable

A page is indexable when every one of these is true at once. It returns HTTP 200. It is not blocked by robots.txt. It carries no noindex in a meta tag or an X-Robots-Tag header. It is the canonical version, or Google chooses it as the canonical among duplicates. And its content renders and carries real value. Break any one of these and the page falls out of the index, even if the other four hold.

Two of these conditions cause most surprises. The canonical condition means a page can be perfect and still not index on its own URL, because Google folded it into a near-identical page and indexed that one instead. Duplication is the usual trigger. Canonicalization → The value condition means a technically flawless page can still be judged not worth keeping. Google renders it, sees nothing a searcher needs, and skips it. That is a content problem wearing a technical costume. Content quality and E-E-A-T →

A page that returns 200 has cleared crawling, not indexing. Indexability is five conditions holding at once, and quality is one of them.

Reading the Page Indexing report

Search Console's Page Indexing report groups your URLs by why they are or are not indexed. The status labels are the diagnosis, and each one points at a different fix. The ones you will meet most:

• Crawled - currently not indexed. Google fetched the page and chose not to keep it. This is almost always a quality or value signal, not a bug. The page is competing for a slot and losing. Raise the quality bar →
• Discovered - currently not indexed. Google knows the URL exists but has not even crawled it yet. This points at crawl budget or perceived low value: the page is low in the queue, often because the site floods Google with URLs or the page looks unpromising from its links.
• Duplicate without user-selected canonical and Duplicate, Google chose different canonical. Google decided another URL is the real version and indexed that one. Align your signals or accept its pick.
• Excluded by noindex tag. The page carries a noindex directive. Intentional or not, that is why it is out.
• Blocked by robots.txt. Crawling is disallowed, so Google could not process the page for indexing.
• Soft 404. The page returns 200 but reads as empty or an error, so Google will not index it.

Others in the same report cover redirects, real 404s, and 5xx server errors. Read the label first, then decide whether the page belongs in the index at all. To inspect a single URL and see its indexed state, canonical, and last crawl, use URL Inspection alongside this report. Search Console reports →

Managing index bloat

Index management runs in two directions. Grow the good and cut the bad. Growing the good means helping strong pages get discovered and kept: link to them from pages Google already crawls often, keep them shallow in the site structure so they are a few clicks from the homepage, and make sure they render with real value. A page nobody links to is hard for Google to find a reason to index.

Cutting the bad is the other half, and it is where most large sites go wrong. Index bloat is a pile of thin, duplicate, or auto-generated URLs that dilute how Google sees the site and waste its attention. Tag pages with one post, filter combinations, printer versions, empty search results, and stale parameter URLs are the usual suspects. For each low-value cluster, decide the outcome: add real content, consolidate into a stronger page, apply noindex to keep it crawlable but out of the index, or remove it and return a proper status. The goal is an index that reflects the pages you actually want ranking, not every URL your platform can generate.

One caution repeated from the crawl chapter: noindex keeps a page out of the index but does not save crawl budget, because Google must fetch the page to read the tag. Index management and crawl management are related but not the same lever. Use noindex to control what gets stored, and use crawl controls for what gets fetched.