ClaudeBot

Anthropic's crawler used to collect content for training and grounding Claude models.

Definition

ClaudeBot is the user-agent string Anthropic uses to crawl the web. It collects publicly available content that may be used to train Claude models and to power retrieval-augmented responses. Like GPTBot, it respects robots.txt directives targeting the `ClaudeBot` user-agent.

Why it matters for AI visibility

Blocking ClaudeBot means Anthropic's systems have less direct access to your canonical brand pages, making it harder for Claude to surface accurate, current information about your products in answers.

GPTBotOpenAI's web crawler that fetches content to train and update its models.

PerplexityBotPerplexity AI's crawler that indexes content for real-time answer generation.

Google-ExtendedGoogle's opt-out user-agent for AI product training, separate from regular search crawling.

robots.txtA plain-text file at the root of your domain that tells crawlers which paths they may or may not fetch.

↗ Checklist: AI crawler accessGPTBot, ClaudeBot, PerplexityBot, Google-Extended, and other AI crawlers need clear permission to fetch important pages.

Check your site

The free scan checks crawler access, robots.txt, sitemap, structured data, and discoverability — and turns the results into a prioritized fix list.

Run the free scan →Back to glossary