1/2/2025

AI Crawler Access: 10-Step Checklist

Stop blocking GPTBot/ClaudeBot/PerplexityBot by accident. A fast technical checklist.

  1. Verify /robots.txt is reachable (200).
  2. Allow agents you want indexed: User-agent: GPTBot / Allow: /
  3. Expose sitemap.xml.
  4. Check CDN bot settings (e.g., Cloudflare Bot Management).
  5. Return fast HTML (TTFB < 600ms). Avoid JS-gated content.
  6. Set canonical URLs and avoid parameter dupes.
  7. Add Organization, FAQ, Product, and Article JSON-LD.
  8. Confirm homepage indexable (no noindex).
  9. Ensure critical folders aren’t disallowed (e.g., /blog).
  10. Re-scan with your AI Visibility Checker and compare.