Free tool

robots.txt generator for AI crawlers

Choose which AI crawlers may read your site and copy the file. The AI companies run separate crawlers for search answers and for training, so you can appear in AI answers without your pages being used to train models.
Start from
Search crawlers: read your pages for AI answers

OAI-SearchBot OpenAI

ChatGPT search answers

PerplexityBot Perplexity

Perplexity's search index

Claude-SearchBot Anthropic

Claude's web search

Googlebot Google

Google Search, AI Overviews, AI Mode and Gemini grounding

Bingbot Microsoft

Bing, Copilot and the services that use Bing's index

Applebot Apple

Siri and Spotlight suggestions

On request: open a page when someone asks

ChatGPT-User OpenAI

ChatGPT opening a link for someone

Perplexity-User Perplexity

Perplexity opening a link for someone

Claude-User Anthropic

Claude opening a link for someone

Training crawlers: collect pages to train models

GPTBot OpenAI

OpenAI model training

ClaudeBot Anthropic

Anthropic model training

Google-Extended Google

Gemini training and grounding in some Google products (not Search)

Applebot-Extended Apple

Apple model training

CCBot Common Crawl

The Common Crawl dataset many models train on

Your robots.txt

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

It replaces the whole file, so compare it with your current one first. It lives at the root of your site: https://yourbusiness.co.uk/robots.txt. Robots.txt is a request that well-behaved crawlers follow, not a lock.

Search crawlers and training crawlers are different

OpenAI, Anthropic and Perplexity each document separate user agents: one that reads pages for their search features, one that opens a page when a person asks, and (for OpenAI and Anthropic) one that collects pages for training. Blocking the training crawler doesn't keep you out of the answers. Blocking the search crawler does. Google works differently: Googlebot covers Search, AI Overviews and AI Mode, and Google-Extended is the token for Gemini training. The details and sources are in our guide to letting ChatGPT crawl your website.

Check the file you have now

Before replacing your robots.txt, run the AI crawler checker to see what it allows today. Many sites block an AI search crawler by accident, through a security plugin or a catch-all rule. Some website builders also manage robots.txt for you, so look for the setting there rather than uploading a file.

Being readable is the start, not the finish. To see whether AI assistants actually recommend you, run the free AI visibility check.