Can ChatGPT, Claude and Perplexity Read Your Site?

Somewhere right now a robot is reading your website. Several, actually. Some of them want to answer a question in ChatGPT. Some want to train the next model. One is there because a person typed your URL into a chat window and said “summarise this”.

Most people who look into this want one thing: block the AI bots. Fair enough. The catch is that each AI company runs several crawlers with different jobs, and blocking the wrong one is easy. You can stop your pages from training models and still show up in AI answers. You can also, with one confident line in robots.txt, disappear from ChatGPT search while the training crawler you meant to block keeps going.

This post covers which crawler does what, what blocking each one actually does, and a 10-minute check for the four places where an AI crawler can be stopped. We ran the check on our own site too.

In short
Each AI company has separate crawlers for search answers, for training and for fetching a page when a person asks. Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended) doesn't remove you from AI answers. Blocking a search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) does. Check four things in order: robots.txt, page rules like noindex and nosnippet, your firewall or CDN, and whether your text is in the HTML without JavaScript.

Three kinds of AI crawler

Every big AI company now splits its robots into roughly three jobs. The names are what you put in robots.txt.

OpenAI's crawler documentation says the switches really are separate: “Each setting is independent of the others”, so “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot”. Anthropic and Perplexity work the same way. Perplexity says PerplexityBot “is not used to crawl content for AI foundation models.”

Google is the odd one out. Its AI Overviews and AI Mode are part of Search, so they use plain Googlebot. Google-Extended isn't even a crawler. Google's crawler documentation says it “doesn't have a separate HTTP request user agent string” and is “used in a control capacity”: it decides whether your pages help train Gemini models and ground answers in the Gemini apps. And it “does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”

Microsoft has no separate Copilot crawler. Bing's webmaster guidelines say Copilot relies on “the same core crawling, indexing, and ranking foundation as traditional search.” If Bingbot can't read you, Copilot can't either.

Which one should you block?

That depends on what you're trying to stop.

If you don't want your content used to train models, block the training tokens: GPTBot, ClaudeBot and Google-Extended. Add CCBot (Common Crawl) if you want to be thorough. You stay in ChatGPT search, Claude, Perplexity and Google's AI features, because those run on the search crawlers.

If you want to stay out of AI answers too, you'd block the search crawlers. OpenAI is blunt about the result: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” Before you do it, look at what AI search is worth to you. Ahrefs measured that AI sends “just 0.1% of total referral traffic”. Small today, and nobody knows how big it gets. Your call.

Here's a robots.txt that blocks training and keeps the answer crawlers in:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

One trap worth knowing. A crawler that has its own group in robots.txt ignores the User-agent: * group completely. So if you write a group for OAI-SearchBot with one Disallow line, it no longer sees any of your general rules. People find this out the hard way.

How common is blocking? Ahrefs looked at about 140 million sites and found that GPTBot “is the most blocked AI bot, with 5.89% of all websites blocking them.” So most sites block nothing. Some on purpose, many probably because nobody ever opened the file.

Does robots.txt actually work?

Mostly, for the crawlers that crawl on their own. OpenAI, Anthropic, Perplexity and Google all say their automatic crawlers follow it. Anthropic even promises that its bots respect anti-circumvention technology: “we will not attempt to bypass CAPTCHAs for the sites we crawl.” Comforting.

The fetchers that act for a person are a different story. OpenAI says of ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.” Perplexity says its user fetcher “generally ignores robots.txt rules.” Google says the same about its user-triggered fetchers. Anthropic is the exception: it says Claude-User follows robots.txt like its other bots.

And then there's the fight. In August 2025 Cloudflare reported that Perplexity used “a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked.” Perplexity answered the same day that Cloudflare had confused it with someone else's traffic, and that Cloudflare's leadership was “either dangerously misinformed on the basics of AI, or simply more flair than cloud.” We can't settle that one for you. What it shows is that robots.txt is a polite sign, not a lock. Cloudflare's own documentation says it plainly: “robots.txt directives are voluntary. Crawlers can ignore them.”

The 10-minute check

An AI crawler has to get through four doors before it can use your page. Check them in this order.

1. Open your robots.txt. Go to yoursite.com/robots.txt. Look for groups that name AI crawlers and remember the trap above: a crawler with its own group ignores the * group. Our free robots.txt tester shows which rule wins for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended.
2. Check the page rules. View the source of a few important pages. noindex keeps a page out of Google Search and its AI features. nosnippet and max-snippet limit what Google can show. Google's guide to AI features lists exactly these controls, and Bing adds two of its own: “NOARCHIVE prevents content from being used in Copilot responses and grounding results”, and NOCACHE limits Copilot to the URL, title and snippet.
3. Check your firewall and CDN. This is the door robots.txt testers can't see. Cloudflare and other CDNs can block AI crawlers at the edge, whatever your robots.txt says, and Cloudflare's defaults have changed more than once since 2025. Open Security → Settings in Cloudflare and look for the AI Crawler and Agent access policies and Bot Fight Mode. Perplexity's own docs warn that “Cloudflare’s Bot Fight Mode can’t be skipped with WAF custom rules.” The only proof of what real crawlers get is your server logs.
4. Look at your page without JavaScript. View the source, or turn JavaScript off in your browser and reload. If the text disappears, most AI crawlers never see it. In Vercel's study of AI crawlers from late 2024, “none of the major AI crawlers currently render JavaScript.” ChatGPT's and Claude's crawlers do download your JavaScript files. They just never run them. Google's Gemini is the exception, because it uses Googlebot.

Our free AI Readiness Checker does doors 1, 2 and 4 in one go for ChatGPT, Claude, Perplexity, Google and Copilot. For door 3 it can tell you when Cloudflare is in front of your site, and whether that Cloudflare belongs to your host (Render and Shopify run their own), but not what Cloudflare tells the real crawlers.

If you're on WordPress

Yoast SEO has a switch for this under Settings → Advanced → Crawl optimization. Yoast's own help page says: “If you’re using Yoast SEO free, you can only block Google AdsBot.” Premium adds GPTBot, CCBot and Google-Extended. Those are all training crawlers, so the switch does what most people want. It doesn't touch the search crawlers, which is probably also what you want.

What DeepSpider checks

A DeepSpider audit records the robots.txt rules for the main AI crawlers, using the same group logic Google uses. Two checks report on them. AI search crawlers blocked in robots.txt covers OAI-SearchBot, PerplexityBot and Claude-SearchBot and is marked Needs review, because plenty of sites block them by accident with one wide rule. AI training crawlers blocked in robots.txt covers GPTBot, ClaudeBot, Google-Extended and CCBot and is information only: blocking them is a business decision, not an SEO error. Neither counts towards the Website Health Score.

The AI Readiness view in each audit shows, per answer engine, whether it can crawl the site: ChatGPT search, Claude, Perplexity, Google AI Overviews and AI Mode, Gemini, Microsoft Copilot, Grok and DeepSeek. Grok and DeepSeek always show “Not verifiable”, because neither publishes a crawler name or IP ranges. We'd rather say we don't know than guess.

The free AI Readiness Checker requests your pages with its own name, DeepSpider, never pretending to be an AI crawler. That's also its limit. If your firewall treats the real GPTBot differently, the checker can't see it.

That's what Bot activity is for. You pick your server log files, your browser reads them, and DeepSpider checks every request that claims to be GPTBot, ClaudeBot, PerplexityBot or Googlebot against the IP ranges each company publishes. Requests from anywhere else are counted as impersonating. IP addresses never leave your browser.

FAQ

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI recommends allowing it if you want to appear there.

Does blocking Google-Extended affect AI Overviews?

No. AI Overviews and AI Mode use Googlebot. Google-Extended only controls Gemini training and grounding in the Gemini apps and Vertex AI.

Will blocking AI crawlers hurt my Google rankings?

Not if you leave Googlebot alone. Google says Google-Extended is not a ranking signal in Search.

How long until a robots.txt change takes effect?

OpenAI says about 24 hours for ChatGPT search, and Perplexity says up to 24 hours. Google says recrawling can take “anywhere from several days to several months”.

We checked our own site

Our robots.txt is two lines: User-agent: * and Allow: /. Everyone gets in, training crawlers included. So door 1 was easy.

We ran the same code as our free AI Readiness Checker against deepspider.ai. All five answer engines came out ready: robots.txt allows them, there's no noindex or nosnippet on the public pages, and the pages have the same amount of text with JavaScript turned off as with it on. Door 2 and door 4, done.

Then came the one warning: “Review Cloudflare’s AI crawler settings”. Fair point, except the Cloudflare in front of deepspider.ai isn't ours. Our own Cloudflare account only handles DNS for the site, so the Cloudflare in those responses sits on our host Render's side, and we can't open its bot settings. Our own tool had told us to check a dashboard we don't have. So we fixed the tool: it now recognises when the Cloudflare belongs to a host like Render or Shopify and tells you to ask the host instead. While we were at it, it also stopped saying that robots.txt may not apply to Claude-User, because Anthropic says Claude-User follows it.

Door 3 is still the one we can't see on our own site. The honest answer is that we think real crawlers get through, and we'd need access logs to prove it. We don't have access logs for the public site yet. Which makes us, for now, a company that sells log analysis and has no logs. We're working on the irony.

Sources