AI readiness guide
How to let ChatGPT and AI crawlers access your website
Short answer
To let AI assistants reach your site, allow the relevant AI crawlers in robots.txt and make sure no meta robots or X-Robots-Tag noindex directive blocks the page. AI search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) are separate from AI training bots (GPTBot, ClaudeBot), so you can allow search discovery while making a separate decision about training.
Search crawlers vs training crawlers
AI crawlers are not all the same. Search-discovery bots such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot fetch pages to answer user queries in AI search. Training-related bots such as GPTBot and ClaudeBot, plus model-usage controls like Google-Extended, are about training and model usage — a separate policy choice.
For most public indie tools and product sites, the common default is to allow AI search discovery. Blocking those bots reduces how often your site can appear in AI answers.
A starting robots.txt policy
In robots.txt you can explicitly allow the search bots you want while keeping normal search engines allowed. Be careful with broad Disallow rules and wildcards that accidentally catch AI search bots. If you want to block training but keep search, allow the search user-agents and disallow the training ones separately.
Always confirm the exact, current user-agent tokens, because AI providers add and rename bots over time. A generator can produce copy-paste snippets for the policy you choose.
Do not forget page-level blockers
robots.txt is not the only place a page can be blocked. A meta robots noindex tag or an X-Robots-Tag: noindex response header will keep a page out of search regardless of a permissive robots.txt. Check both before assuming a page is discoverable.
After making changes, verify them: check which named bots are actually allowed or blocked for a specific URL rather than assuming the policy is correct.
Tools for this
Or run a full AI readiness check to see crawler access, llms.txt, and schema in one report.
FAQ
Should I block AI crawlers?
For public tools and SEO-focused sites, the usual default is to allow AI search discovery. Blocking is an intentional policy choice that reduces AI-search visibility.
Can I allow ChatGPT search but block model training?
Often yes. Some providers offer separate user agents for search versus training, so you can allow search discovery and disallow training bots in robots.txt.
Why is my page still not crawlable after editing robots.txt?
Check for a meta robots noindex tag or an X-Robots-Tag noindex header. Either one blocks the page even when robots.txt allows it.
Related guides
Why isn't my site showing up in ChatGPT or AI Overviews?
Common technical reasons a site is missing from ChatGPT, Perplexity, and Google AI Overviews — blocked crawlers, noindex, thin content, no schema — and how to check them.
GEO vs AEO vs SEO: what's the difference?
GEO, AEO, and SEO explained simply: SEO ranks links, AEO wins the direct answer, and GEO earns mentions inside AI-generated responses. How they overlap and what to do.