Checklist / Crawl Access
Required·Crawl Access
AI crawler access
GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and other AI crawlers need clear permission to fetch important pages.
Why AI systems care
If answer engines cannot crawl the pages that define your brand, they are more likely to rely on competitors, stale references, or third-party summaries.
How to check it manually
- 01Open /robots.txt and look for rules that block AI crawler user agents.
- 02Confirm product, category, pricing, documentation, and comparison pages are not disallowed.
- 03Run a live scan to verify crawler-specific access rules.
How to fix it
- 01Remove accidental disallow rules for public pages that should be discoverable.
- 02Keep private, admin, cart, and account areas blocked.
- 03Document crawler policy decisions so marketing and engineering do not fight each other later.
Check your site
The free report checks the highest-impact crawler, technical, and discoverability signals and turns them into a prioritized fix list.