robots.txt
The traditional crawler-control file. Tells search engines and AI crawlers which paths they can fetch.
Two common mistakes: blocking AI crawlers by default (opts you out of citations) and forgetting to allow new bot user-agents like ClaudeBot or PerplexityBot. Audit yours quarterly. A wide-open robots.txt is almost always the right default unless you have specific legal or commercial reasons to block.
See where you stand
Free AI visibility check across ChatGPT, Perplexity, Gemini, and Claude. Results in under a minute.