What does “AI ready” mean?
AI search engines such as ChatGPT search, Claude, Perplexity, Google AI Overviews and Microsoft Copilot answer questions with text drawn from web pages. Before a page can appear in such an answer, three things must be true. The engine’s crawler must be allowed to request it. The response must contain the content in a form the crawler can read. And the page must not tell the search engine to keep it out of its index or its snippets.
None of this guarantees a citation. Which pages an AI system chooses depends on the question, the competition and the system itself. What a checker can do honestly is find the things that stop AI use for certain — a blocked crawler, a noindex, an empty HTML shell — and show you the exact line that causes each one.
OpenAI, Anthropic, Google, Apple and others use separate crawlers for search and for model training. Blocking GPTBot or Google-Extended keeps your content out of training, but does not remove you from ChatGPT search or Google AI Overviews. This checker never counts a training opt-out against you.
How this tool works, step by step
Read robots.txt
The checker reads your robots.txt once and evaluates it for 14 AI-related crawlers, using each crawler’s own user-agent group, exactly as the crawler would. It shows the rule that decided each result. It also honours the rules for DeepSpider itself: if robots.txt asks DeepSpider not to fetch a page, we do not.
Request your page as DeepSpider
It requests the page with DeepSpider’s own user agent, follows up to three redirects on the same website and records the status and headers. It recognises Cloudflare and SiteGround challenge pages from their headers instead of guessing from page size.
Read the HTML source
From the HTML it reads robots meta tags and X-Robots-Tag headers for Google and Bing, snippet controls, the text available without JavaScript, title, description, language, headings, canonical tags and JSON-LD.
Check four more pages
It reads your XML sitemap and checks up to four more pages from different sections, because templates often differ between the homepage, articles and product pages.
Test a missing page
It requests one address that cannot exist and checks that it returns 404. AI assistants request non-existent URLs far more often than Google does.
Summarise per engine
For each engine it combines that engine’s crawler rule with the directives that engine honours. Every reason is shown. Nothing is generated by AI and nothing is scored with hidden weights.
We deliberately do not pretend to be GPTBot or ClaudeBot. Many firewalls verify these crawlers by their IP addresses, so a request that only borrows their name is treated as an impostor and proves nothing about what the real crawler receives.
How to read the report
Nothing stops AI use
The engine’s crawler is allowed and no directive or missing content limits the page. It may still not be cited — that depends on the answer being written.
Partly restricted
Some sampled pages are blocked, snippets are capped, or the HTML holds little text without JavaScript. The engine can use the page only partly.
AI use is prevented
robots.txt blocks the engine’s crawler, the page is noindex, or a directive such as nosnippet or Bing’s NOARCHIVE forbids AI use. The report names the line.
Not verified
robots.txt or the page could not be read — often a firewall challenge. Your firewall may treat the real AI crawlers differently; check its logs.
Since July 2025 Cloudflare can block AI crawlers at the edge, whatever robots.txt says. If your site is served through Cloudflare, open AI Crawl Control in the dashboard and check that the search crawlers you want are allowed.
What helps — and what does not
Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot if you want to appear in their AI answers. Decide on training crawlers separately.
Most AI crawlers do not run JavaScript. Use server-side rendering or pre-rendering so the main text, links and canonical are in the HTML response.
Google uses only indexed, snippet-eligible pages in AI Overviews and AI Mode. Avoid noindex and nosnippet where you want visibility; use data-nosnippet for specific sections only.
Answer missing URLs with 404 or 410 and redirect moved pages, so assistants that guess URLs do not land on soft 404s.
Google says AI search needs no special files or markup. Search does not use llms.txt, and controlled tests found no citation effect from adding schema or “chunking” content.
Only your server logs or CDN dashboard show what the real AI crawlers receive. Check them after every firewall or robots.txt change.
Frequently asked questions
Does a ready result mean ChatGPT will recommend my website?
No. It means nothing we could observe stops AI search from fetching, reading and quoting the page. Whether an assistant cites you depends on the question, other sources and the system itself. This checker does not ask AI systems about your brand.
Should I block GPTBot or other training crawlers?
That is a business decision about training use, not about search visibility. Blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot; Google-Extended does not affect Google Search or AI Overviews. The report shows training crawlers for information only.
Why is robots.txt not the whole story?
Firewalls and CDNs can block crawlers that robots.txt allows. Cloudflare in particular can block AI crawlers at the edge. The checker shows what DeepSpider’s own request received, but only your logs show what the real crawlers get.
Why doesn’t the checker test with the real GPTBot user agent?
A request that only borrows a crawler’s name comes from our servers, not the crawler’s. Firewalls that verify crawlers by IP address treat it as an impostor, so the result would be misleading in either direction.
Do I need an llms.txt file?
Google says Search does not use llms.txt, and an Ahrefs study of server logs found that 97% of llms.txt files received no requests at all. The report mentions whether you have one, but it does not affect any result.
Does structured data help AI search?
Google says structured data is not required for AI features, and a controlled Ahrefs test found no measurable effect on AI citations. It still helps Google rich results, so the checker validates any JSON-LD you have without rewarding you for adding more.
Why does my JavaScript website look empty?
The checker reads the HTML the server sends, like most AI crawlers. If your content is built in the browser by JavaScript, those crawlers may see an almost empty page. Google renders JavaScript, other AI crawlers mostly do not.
Is the check free and private?
Yes. There is no signup or email gate. We do not publish a report page for the websites you check. Repeat checks of the same URL within ten minutes reuse the previous result.
Where to check the underlying guidance
Read the crawler documentation from OpenAI, Anthropic and Perplexity, Google’s AI features and your website and common crawlers, and Cloudflare’s AI crawler controls. The Vercel and MERJ study The rise of the AI crawler measured that the major AI crawlers do not run JavaScript.