Log File Analysis and Crawl Monitoring

Every report about how Google crawls you is a summary or a sample. Your server logs are the record. They show exactly which URLs bots requested, when, how often, and what they got back. For diagnosing crawl behavior on a large site, logs are the only ground truth, and learning to read them turns guesses into evidence.

What server logs are

A server access log records every request the server answered, one line per request. Each line carries the client IP, a timestamp, the method and URL requested, the HTTP status returned, the bytes sent, the referrer, and the user-agent string. Read across those fields and you can see precisely who asked for what and how the server responded.

This is ground truth in a way no dashboard is. Search Console's Crawl Stats report is genuinely useful, but it is sampled and aggregated, a representative slice rather than the full record. When you need to know exactly how Googlebot moved through your site, whether it fetched a particular URL, how often, and what code it hit, only the raw logs answer without approximation.

Logs are also the one crawl record you own outright. Search Console shows you Google's view, filtered through Google's tooling and retention limits. The access log is your server's own account of every request, including the AI and third-party crawlers Search Console never reports on at all. That makes it the right place to answer questions no Google tool will, like which non-Google bots are hammering your site and what they take.

What logs reveal

Reading logs surfaces problems no other tool shows cleanly:

• Crawl-budget waste. Which URL patterns eat the most Googlebot requests. If half of Google's visits go to filter and parameter URLs, the logs prove it and rank the offenders by hit volume. Crawl budget →
• Orphan pages Google crawls. URLs bots keep fetching that your site no longer links to internally, a sign of stale links, old sitemaps, or leftover paths.
• The status codes bots actually hit. Not what you think you return, but the real mix of 200s, redirects, 404s, and 5xx errors Googlebot experiences.
• Pickup speed. How quickly bots discover and crawl new content after you publish it.
• Spoofed bots. Requests claiming to be Googlebot in the user-agent that are not, which you can only catch by verifying rather than trusting the string.

The analysis procedure

The workflow is repeatable. Collect logs over two to four weeks, longer for very large sites, so you capture Google's slower recrawl cycles and not just a busy afternoon. Filter to verified bot traffic, discarding spoofed and human requests, so you are analyzing real crawl behavior. Load the filtered logs into an analyzer built for the job: Screaming Frog's Log File Analyser for most sites, or Splunk, the ELK stack, or BigQuery at larger scale.

Then aggregate. Group Googlebot requests by URL pattern and by status code, and two lists fall out: the top-crawled useless URLs (parameter noise, dead patterns, endless facets) and the rarely-crawled valuable URLs (money pages Google barely touches). Cross-reference a fresh site crawl to find orphans, pages in your crawl that bots never hit, and pages bots hit that your crawl never found. Fix the biggest waste first, then re-sample to confirm the change took. This is also how you confirm a robots.txt rule actually took effect, by watching the blocked pattern's hits fall to zero. Confirming robots rules →

Search Console's crawl stats are sampled; the server logs are the record. When you need to know exactly how bots crawl your site, the logs are the only source that does not approximate.

Verify the bot, never trust the string

The user-agent string is trivial to fake, and scrapers spoof Googlebot constantly to slip past rate limits. Never treat a request as Googlebot because its user-agent says so. Verify it. Do a reverse DNS lookup on the IP address and confirm it resolves to a googlebot.com or google.com host, then do a forward lookup on that hostname and confirm it points back to the same IP. That reverse-then-forward check, or matching the IP against Google's published ranges, is the only reliable proof.

Apply the same rigor to AI crawlers. Their fetchers are worth watching in your logs to see which pages the answer engines actually pull, and their identities can be spoofed too, so verify them by IP and published ranges rather than the user-agent. Verifying AI-bot hits →

Logs close the technical loop: they confirm whether every rule in the preceding chapters landed the way you intended. With crawling, indexing, and infrastructure understood and measurable, the next study moves to how fast those pages load for real users. Core Web Vitals next →