Guide

How to allow ChatGPT to crawl your website

Updated · 11 min read · By the SearchCrow team

To let ChatGPT read and cite your website, allow OpenAI's OAI-SearchBot crawler in your robots.txt file and make sure your host or CDN is not blocking OpenAI's published IP addresses. OpenAI says that is what makes a site "eligible for inclusion" in ChatGPT search (OpenAI Help Center, 2026). GPTBot, the crawler that collects training data, is a separate choice: blocking it does not remove you from ChatGPT's search answers.

Key takeaways

  • OpenAI runs three crawlers. OAI-SearchBot is the one that decides whether you appear in ChatGPT search answers.
  • Allowing search and allowing training are independent decisions for OpenAI and Anthropic. Google-Extended is the exception, because it also controls Gemini grounding.
  • Blocks also happen in a CDN or host setting, or in a theme that hides text in JavaScript, not only in robots.txt.
  • OpenAI and Perplexity say robots.txt changes take about 24 hours to apply; Google caches robots.txt for up to 24 hours.

Which OpenAI crawler does what?

OpenAI documents three user agents, and only one of them controls whether you show up in ChatGPT search (OpenAI, 2026).

CrawlerWhat OpenAI says it doesEffect of blocking itObeys robots.txt?
OAI-SearchBot"OAI-SearchBot is for search." It surfaces websites in ChatGPT's search featuresOpted-out sites "will not be shown in ChatGPT search answers", though they can still appear as navigational linksYes
ChatGPT-UserVisits a page when a user's question needs it, and interacts with external apps via GPT Actions. "Not used for crawling the web in an automatic fashion"Not used to decide whether you appear in searchBecause a person initiated the request, "robots.txt rules may not apply"
GPTBotCrawls content that may be used to train OpenAI's models"Indicates a site's content should not be used" for training. No effect on searchYes

Two further details from the same page: "Each setting is independent of the others", so you can allow OAI-SearchBot and disallow GPTBot; and if both are allowed, one crawl may serve both purposes. Each crawler publishes its IP addresses as a JSON file (openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json), which your host can use to check a firewall is not blocking OpenAI.

The other assistants follow the same pattern

Perplexity and Anthropic also split search from training, so the decision below applies to them too.

PlatformSearch or answer crawler (allow)User-initiated fetcherTraining crawler (your choice)
OpenAIOAI-SearchBotChatGPT-UserGPTBot
PerplexityPerplexityBot, which is "not used to crawl content for AI foundation models"Perplexity-User, which "generally ignores robots.txt rules"None documented
AnthropicClaude-SearchBot; blocking it "prevents our system from indexing your content for search optimization"Claude-User; blocking it "prevents our system from retrieving your content in response to a user query"ClaudeBot
GoogleGooglebot (Google Search, AI Overviews and AI Mode)None documentedGoogle-Extended, a robots.txt token with no separate user agent, which also controls grounding in Gemini Apps

Sources: Perplexity, 2026; Anthropic, 2026; Google Search Central, 2026. See how AI assistants choose businesses for where each gets its answers.

Should I allow AI crawlers at all?

Allow the search crawlers; decide separately about training.

The search crawlers are the ones that bring customers: if OAI-SearchBot cannot read your site, ChatGPT cannot quote your prices, hours or service area, and will describe a competitor instead. Blocking the training crawlers (GPTBot, ClaudeBot) is a legitimate choice; OpenAI and Anthropic both say it does not affect their search features.

Google-Extended is the one to leave alone unless you have thought it through. Google says it controls both training and "grounding" in Gemini Apps, and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google Search Central, 2026). Blocking it may reduce how Gemini uses your content; Google Search and AI Overviews are unaffected.

How do I check my current robots.txt?

Type your domain followed by /robots.txt into a browser, for example https://www.example.co.uk/robots.txt. Google's specification says the file must sit in the top-level directory and applies only to that host, protocol and port (Google Search Central, 2026). You will see one of three things:

  1. A 404 "not found" page. You have no robots.txt. Google treats this as no crawl restrictions, so every crawler is allowed. You may still be blocked elsewhere (see below).
  2. Only User-agent: * and a few Disallow lines for admin folders. The normal state for most sites; AI crawlers are allowed.
  3. OAI-SearchBot, PerplexityBot, Claude-SearchBot or Google-Extended with Disallow: / under it. Someone, or a plugin, has opted you out. Read on.

Two rules from Google's specification catch people out (Google Search Central, 2026):

  • User-agent names are case-insensitive; paths in Disallow lines are case-sensitive.
  • A crawler follows the "group with the most specific user agent that matches" it, and "user agent specific groups and global groups (*) are not combined". A group for OAI-SearchBot ignores your * rules, so repeat any folders you want kept private.

Google describes robots.txt as telling crawlers "which URLs the crawler can access on your site" and says "it is not a mechanism for keeping a web page out of Google" (Google Search Central, 2026). Compliance is voluntary, but every crawler in this guide follows it.

What should my robots.txt say?

Copy this, replace the domain and private folders, and upload it to your site root. It allows every documented AI search crawler, optionally opts out of training, and keeps an admin folder private. Or build one by choosing each crawler in our free robots.txt generator.

# AI search and answer crawlers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Googlebot
User-agent: bingbot
Disallow: /admin/

# AI model training: optional opt-out. Delete this group to allow training.
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.co.uk/sitemap.xml

Notes:

  • Several User-agent lines above one set of rules form one group.
  • To allow training too, delete the middle group. To block Google's training as well, add User-agent: Google-Extended to it, accepting the Gemini trade-off above.
  • Never add Disallow: / under User-agent: * to "keep bots out". That blocks Googlebot and bingbot too.
  • Anthropic also supports the non-standard Crawl-delay directive if its crawlers are hitting your server too hard (Anthropic, 2026).

Where to edit it on common platforms

  • Wix: go to SEO & GEO, then Tools and settings, then Robots.txt Editor. Wix says the default "allows search engine bots to access all of your site's pages" and that editing it "is an advanced feature" its support team cannot help with (Wix, 2026).
  • Squarespace: you do not edit the file directly. Under Settings, then Crawlers, a checkbox labelled "Block known artificial intelligence crawlers" is unchecked by default "because we don't want to potentially impact your site's traffic". Ticking it adds rules for around 26 bots including GPTBot, ClaudeBot and Google-Extended (Squarespace, 2026). Leave it unticked if you want ChatGPT to find you; it cannot be applied to only some pages.
  • Shopify: every store has a default robots.txt, customised by adding a robots.txt.liquid template in the theme code editor. Blocking AI crawlers there affects only open-web discoverability; product data sent to agentic storefronts such as ChatGPT is managed separately (Shopify, 2026). See ChatGPT shopping for small online shops.
  • WordPress.com: under Settings, then Reading, the Public option has a "Prevent third-party sharing" checkbox that "adds known AI bots to the 'disallow' list" in robots.txt; WordPress.com notes "it is up to AI platforms to honor this request" (WordPress.com, 2026). The article does not name the bots, so if you tick it, open your robots.txt afterwards and check OAI-SearchBot is not listed.

Where else can my site be blocking AI crawlers?

Robots.txt is the first place to look, but it is not the only one. Three other layers block crawlers without touching that file.

1. Cloudflare and other CDNs or firewalls

OpenAI asks site owners to "confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses" (OpenAI Help Center, 2026). Cloudflare, the most common CDN on small sites, has four relevant settings, all documented:

  • AI Crawler and Agent access policies (Security, then Settings, filter by Bot traffic). AI bots are grouped into Search ("crawlers that collect or index your content to answer questions about it later"), Agent ("chat fetch bots and browser-use agents") and Training, each set to Allow, "Block on pages with ads" or Block. Cloudflare says that "for new domains, Search, Agents, and Training use Allow (do not block) by default" and that the defaults can be changed after onboarding (Cloudflare, 2026). If Search is set to Block, OAI-SearchBot is stopped at the edge whatever your robots.txt says. Set Search and Agent to Allow; set Training as you prefer.
  • Bot Preference Sync / managed robots.txt. When enabled, Cloudflare prepends its own directives to your robots.txt, including Disallow: / for training crawlers such as GPTBot, ClaudeBot and Google-Extended. Cloudflare says this "doesn't technically block access" (Cloudflare, 2026). It explains AI rules in your robots.txt that you never wrote.
  • AI Crawl Control (all plans) lists which AI crawlers have visited your domain and lets you set Allow or Block per crawler (Cloudflare, 2026). It is the quickest way to see whether OAI-SearchBot has been reaching you.
  • Bot Fight Mode (Free plan) challenges traffic matching known bot patterns, and Cloudflare says you "cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules"; if it causes problems, it can be turned off in Security Settings (Cloudflare, 2026). Cloudflare's Verified bots directory lists OAI-SearchBot and Perplexity-User as recognised bots (Cloudflare, 2026); still, check Security Analytics for challenged AI traffic if ChatGPT cannot see pages you know are allowed.

Other CDNs, security plugins and anti-scraping services have similar toggles. Search their settings for "AI", "bot" or "crawler".

2. Your hosting firewall

Some hosts block user agents or IP ranges at the server level. Ask yours whether they block AI crawlers or the OpenAI, Perplexity or Anthropic IP ranges, and point them to the vendors' published JSON lists. Anthropic warns that IP blocking "may not work correctly or persistently guarantee an opt-out" because it stops its bots reading your robots.txt (Anthropic, 2026); the reverse also holds, so an IP block can hide a site that robots.txt says is open.

3. Pages that only work with JavaScript

A crawler can be allowed in and still read a blank page. If your services, prices and phone number are drawn in by JavaScript after the page loads, some crawlers never see them. Google notes that "not all bots can run JavaScript" and that "server-side or pre-rendering is still a great idea" (Google Search Central, 2026). OpenAI, Perplexity and Anthropic do not document whether their crawlers run JavaScript, so do not rely on it.

Check: open your home page, right-click, choose View page source, and search for your business name and a price. If they are missing, the text lives in scripts, and the fix is a theme or build setting (server-side rendering or pre-rendering) your developer will recognise.

How can I test whether AI crawlers can reach my site?

There is no "fetch as OAI-SearchBot" button. The practical checks below all use tools the vendors document.

  1. robots.txt in a browser. Load /robots.txt and read it against the table above. It is the one check that covers OpenAI, Perplexity and Anthropic together.
  2. Search Console's robots.txt report. Shows the files Google found, their fetch status, last check and any parsing errors, with a "request a recrawl" option after a fix (Google Search Console Help, 2026). Parsing errors affect every crawler, not only Google.
  3. Search Console's URL Inspection live test. Choose Test live URL and look at "Crawl allowed?" and "Page fetch"; View tested page shows the HTML, headers and a screenshot as Google rendered it (Google Search Console Help, 2026). A firewall that blocks Google usually blocks others too.
  4. Your CDN's crawler log. Cloudflare's AI Crawl Control Crawlers tab shows which AI crawlers accessed your domain and the action taken (Cloudflare, 2026). OAI-SearchBot listed with "Allow" is the nearest thing to confirmation.
  5. Ask ChatGPT. With search on, ask a question only your site answers (an illustrative example: "What does [your business] charge for a boiler service in Leeds?") and open Sources. A citation of your page proves it was read. No citation proves nothing on its own; ChatGPT may simply have preferred another source.

How long do changes take to apply?

Roughly a day for the crawlers that publish a figure.

VendorDocumented timing
OpenAIAfter a robots.txt update "it can take ~24 hours" for search changes to apply (OpenAI, 2026)
Perplexity"It may take up to 24 hours for our systems to reflect changes" (Perplexity, 2026)
GoogleCaches robots.txt "for up to 24 hours", updating "the cached version every 24 hours"; use "Request a recrawl" in the robots.txt report to hurry it along (Google Search Central, 2026)
AnthropicNo figure published

Those figures cover the crawler noticing the change. Being recommended is separate: the pages then have to be worth citing. Our AI visibility checklist covers what to put on them, and why doesn't ChatGPT recommend my business covers the other reasons a crawlable site is left out.

What should I do this week?

  1. Open /robots.txt and fix any Disallow: / under a search crawler.
  2. In your CDN or host, set AI Search and Agent policies to Allow.
  3. View page source and confirm your services, prices and phone number are in the HTML.
  4. Verify your site in Search Console and read the robots.txt report.
  5. Wait a day, then ask ChatGPT a question your site answers and check Sources.

How SearchCrow helps

SearchCrow's free check reads your website the way AI crawlers see it, flags robots.txt rules that shut out AI crawlers and pages it cannot read, then asks ChatGPT and Perplexity five questions a customer might ask and shows whether you are named. About 30 seconds, no card. Everything in this guide works without it; the check saves repeating the steps each month.

FAQ

Which OpenAI crawler do I need to allow to appear in ChatGPT answers? OAI-SearchBot. OpenAI says it is the crawler for search, and that sites which opt out of it will not be shown in ChatGPT search answers. GPTBot is a separate crawler for model training, and ChatGPT-User fetches a page when a person's request needs it. OpenAI says each setting is independent, so you can allow OAI-SearchBot and still block GPTBot.

Can I block GPTBot but still appear in ChatGPT search? Yes. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as separate user agents and says each setting is independent of the others. Disallow GPTBot in robots.txt and leave OAI-SearchBot allowed. The one caveat OpenAI gives is that if both are allowed, a single crawl may be used for both purposes.

How do I check whether my site is blocking AI crawlers? Open yourdomain.co.uk/robots.txt in a browser and look for Disallow: / under OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot or bingbot. Then check your CDN or host for AI-bot or bot-fight settings; Cloudflare's AI Crawl Control shows which AI crawlers it has blocked. Finally, view your page source and confirm your services and contact details are in the HTML.

Does Cloudflare block ChatGPT by default? Cloudflare's documentation says that for new domains its Search, Agents and Training policies all default to Allow, and that you can change them after onboarding under Security, Settings, AI Crawler and Agent access policies. Check that page rather than assuming: a Block setting on Search would stop OAI-SearchBot regardless of what your robots.txt says.

How long after fixing robots.txt will ChatGPT see my site? OpenAI says robots.txt changes can take around 24 hours to apply to its crawlers, and Perplexity gives the same figure. Google caches robots.txt for up to 24 hours and offers a request-a-recrawl option in Search Console's robots.txt report. Anthropic does not publish a figure. Being crawlable is the first step; being recommended still depends on what the page says.

Sources

Free check

Does ChatGPT recommend your business?

Free, in under a minute. You’ll see whether AI assistants mention you, who they recommend instead, and your top 3 fixes.

We’ll read your website for your business details, so there’s less to type.

Free, no card needed. We ask ChatGPT and Perplexity 5 questions your customers might ask, and check your website. We use your email to send you your results and a few short follow-ups about them, which you can stop with one click; see our privacy notice.