Checklist / Crawl Access
Required·Crawl Access
robots.txt
A valid robots.txt file gives crawlers a predictable policy for what they may request.
Why AI systems care
Malformed or overly broad rules can make your most useful pages invisible to search and AI systems before content quality is even considered.
How to check it manually
- 01Visit /robots.txt and confirm it returns 200 or a clear empty policy.
- 02Check for broad Disallow: / rules.
- 03Confirm sitemap locations are declared when available.
How to fix it
- 01Keep rules simple and specific.
- 02List sitemap URLs in the file.
- 03Avoid blocking public content sections used for discovery and evaluation.
Check your site
The free report checks the highest-impact crawler, technical, and discoverability signals and turns them into a prioritized fix list.