AI crawlers
User-agents that pull content for AI training or live retrieval. The major ones: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, anthropic-ai, CCBot.
Some run continuously to keep retrieval indexes fresh (PerplexityBot, ChatGPT-User). Others run periodically for training (GPTBot, CCBot). Block them and you opt out of being cited. Allow them and ensure they can crawl your important pages quickly, with proper schema and clean URLs.
See where you stand
Free AI visibility check across ChatGPT, Perplexity, Gemini, and Claude. Results in under a minute.