Free tool
robots.txt generator for AI crawlers
Your robots.txt
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
It replaces the whole file, so compare it with your current one first. It lives at the root of your site: https://yourbusiness.co.uk/robots.txt. Robots.txt is a request that well-behaved crawlers follow, not a lock.
Search crawlers and training crawlers are different
OpenAI, Anthropic and Perplexity each document separate user agents: one that reads pages for their search features, one that opens a page when a person asks, and (for OpenAI and Anthropic) one that collects pages for training. Blocking the training crawler doesn't keep you out of the answers. Blocking the search crawler does. Google works differently: Googlebot covers Search, AI Overviews and AI Mode, and Google-Extended is the token for Gemini training. The details and sources are in our guide to letting ChatGPT crawl your website.
Check the file you have now
Before replacing your robots.txt, run the AI crawler checker to see what it allows today. Many sites block an AI search crawler by accident, through a security plugin or a catch-all rule. Some website builders also manage robots.txt for you, so look for the setting there rather than uploading a file.
Being readable is the start, not the finish. To see whether AI assistants actually recommend you, run the free AI visibility check.