PerplexityBot

Perplexity AI's crawler that indexes content for real-time answer generation.

Definition

PerplexityBot crawls the web on behalf of Perplexity AI, an answer engine that cites sources in its responses. Unlike training-focused crawlers, PerplexityBot is primarily used for real-time retrieval — meaning the content it indexes may be included as a citation in a live answer within hours of being crawled.

Why it matters for AI visibility

Perplexity shows citations prominently in its answers. Allowing PerplexityBot and ensuring crawlable, well-structured pages directly increases the chance of appearing as a named source in buyer-intent queries.

GPTBotOpenAI's web crawler that fetches content to train and update its models.

ClaudeBotAnthropic's crawler used to collect content for training and grounding Claude models.

Google-ExtendedGoogle's opt-out user-agent for AI product training, separate from regular search crawling.

robots.txtA plain-text file at the root of your domain that tells crawlers which paths they may or may not fetch.

↗ Checklist: AI crawler accessGPTBot, ClaudeBot, PerplexityBot, Google-Extended, and other AI crawlers need clear permission to fetch important pages.

Check your site

The free scan checks crawler access, robots.txt, sitemap, structured data, and discoverability — and turns the results into a prioritized fix list.

Run the free scan →Back to glossary