//Crawler
JustSEO Crawler
Identify and whitelist the JustSEO crawler on your firewall or web server.
Our crawler helps analyze your website for SEO issues. Here is everything you need to know about identifying and whitelisting it.
User-Agent String
JustSEO-Crawler/1.0 (+https://justseo.ai/crawler)
This User-Agent identifies all requests from JustSEO's crawling infrastructure.
Crawler behavior
- Request rate: our crawler respects rate limits and waits between requests to avoid overwhelming your server.
- Robots.txt: we respect robots.txt directives. Add rules for "JustSEO-Crawler" to control access.
- Timeout: each request has a 30-second timeout to prevent hanging connections.
- Redirects: we follow up to 10 redirects and track the full redirect chain.
Whitelisting guide
Add a rule for the "JustSEO-Crawler" user agent on your firewall or web server.
# robots.txt - allow full access for JustSEO Crawler
User-agent: JustSEO-Crawler
Allow: /- Cloudflare: Security -> WAF -> Custom Rules. Create a rule with action "Skip" and the expression http.user_agent contains "JustSEO-Crawler".
- AWS WAF: add a rule that matches the User-Agent header containing "JustSEO-Crawler" and set the action to "Allow".
- Nginx / Apache: match the User-Agent "JustSEO-Crawler" and skip rate limiting or allow access.
Security and trust
JustSEO maintains standard security trust signals for transparency:
- security.txt - Responsible disclosure policy (RFC 9116)
- Privacy Policy - how we handle your data
- Terms of Service - usage terms and conditions
Need help?
If you are having issues with our crawler or need to report abuse, contact support.