robots.txt

What should your robots.txt do for AI crawlers?

robots.txt sits at the root of your domain and lists rules per user-agent for which paths a crawler may or may not fetch. It has governed search crawling for decades, but its role has expanded: the same file now controls whether AI crawlers can reach your content, which makes it a direct lever on your AI visibility.

Two mistakes are common and costly. The first is blocking AI crawlers by default, often inherited from a security template, which silently opts you out of being cited in AI answers. The second is forgetting to allow newer bot user-agents like ClaudeBot or PerplexityBot, so you stay invisible to the fastest-growing engines without realizing it.

Unless you have a specific legal or commercial reason to restrict access, a wide-open robots.txt is almost always the right default, and it is worth auditing quarterly as new bots appear. GrowthManager reviews robots.txt on every site and keeps the major AI crawlers explicitly allowed.

Use specific Allow rules per user-agent rather than a single broad permissive Disallow. That keeps the file readable, makes intent obvious to anyone reviewing it later, and gives you a clean audit trail if you ever need to revoke access for one engine without affecting the others. Pair the file with a robots-meta-tag policy on the pages themselves so an over-eager security tool that strips robots.txt cannot accidentally re-open paths you meant to keep behind authentication. Layer the two and you get defense in depth with no daily maintenance cost.