JustSEO Crawler

Our crawler helps analyze your website for SEO issues. Here is everything you need to know about identifying and whitelisting it.

User-Agent String

JustSEO-Crawler/1.0 (+https://justseo.ai/crawler)

This User-Agent identifies all requests from JustSEO's crawling infrastructure.

Crawler behavior

• Request rate: our crawler respects rate limits and waits between requests to avoid overwhelming your server.
• Robots.txt: we respect robots.txt directives. Add rules for "JustSEO-Crawler" to control access.
• Timeout: each request has a 30-second timeout to prevent hanging connections.
• Redirects: we follow up to 10 redirects and track the full redirect chain.

Whitelisting guide

Add a rule for the "JustSEO-Crawler" user agent on your firewall or web server.

# robots.txt - allow full access for JustSEO Crawler
User-agent: JustSEO-Crawler
Allow: /

• Cloudflare: Security -> WAF -> Custom Rules. Create a rule with action "Skip" and the expression http.user_agent contains "JustSEO-Crawler".
• AWS WAF: add a rule that matches the User-Agent header containing "JustSEO-Crawler" and set the action to "Allow".
• Nginx / Apache: match the User-Agent "JustSEO-Crawler" and skip rate limiting or allow access.

Security and trust

JustSEO maintains standard security trust signals for transparency:

• security.txt - Responsible disclosure policy (RFC 9116)
• Privacy Policy - how we handle your data
• Terms of Service - usage terms and conditions

Need help?

If you are having issues with our crawler or need to report abuse, contact support.