Free tool

Is your website blocking AI crawlers?

Enter your website and we read its robots.txt the way ChatGPT, Perplexity, Claude, Google and Bing's crawlers do, then show which of them may read your site and the lines that would let the blocked ones in.

We read the site’s robots.txt and homepage once. Nothing is stored.

What this checks, and what it doesn’t

It reads robots.txt and applies each crawler’s rules the way crawlers do: a group naming the crawler beats the catch-all * group, and the longest matching rule wins. It also fetches your homepage once and reports the status code, because a firewall or bot check can refuse crawlers even when robots.txt allows them.

It can’t tell you whether a CDN challenge is blocking a specific crawler, or whether a page only works with JavaScript. The guide to allowing AI crawlers covers those checks, and the free AI visibility check reads your whole site the way the assistants do and asks them five customer questions.

Which crawlers matter?

The search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) decide whether an assistant can find and cite you. The training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) only collect text for model training; OpenAI, Anthropic and Google each document that blocking them does not remove you from their search features. See how AI assistants choose businesses for what each vendor says. To write a new file crawler by crawler, use the robots.txt generator.