AI crawlers

Should you allow or block AI crawlers?

They split into two jobs. Some run continuously to keep a retrieval index fresh, such as PerplexityBot and ChatGPT-User, which is what feeds live answers. Others run periodically to gather training data, such as GPTBot and CCBot. The distinction matters because blocking the retrieval crawlers has an immediate cost: you opt out of being cited in answers right now.

For almost every business that wants AI visibility, the right default is to allow them and then make sure they can actually do their job: reachable important pages, clean URLs, fast responses, and proper schema so extraction is accurate. Blocking by accident, usually through an over-broad robots.txt rule, is one of the most common and most expensive technical mistakes we find.

There are legitimate reasons to block, such as licensing strategy or protecting proprietary content, but those should be deliberate decisions, not defaults inherited from a template. GrowthManager audits crawler access on every site and explicitly grants the engines that matter while keeping a record of anything intentionally excluded.

Server logs are the cleanest way to verify a crawler is actually reading your site. Filter for the major AI user-agents and look for sustained, recurring fetches against your high-signal URLs. If GPTBot only ever pulls your sitemap and never your product pages, something in your robots.txt or your CDN rules is silently filtering it. The same applies to ClaudeBot and PerplexityBot: presence in your logs against the pages you actually want quoted is the difference between an opt-in policy on paper and one that is working in practice.