What robots.txt does — and what it does not
robots.txt is a plain text file at the root of a host, such as https://example.com/robots.txt. It is split into groups. Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules for URL paths. The format is standardised in RFC 9309 and documented by Google, Bing and the AI companies for their own crawlers.
robots.txt controls crawling, not indexing. A URL that robots.txt blocks can still appear in search results if other pages link to it, usually without a description. To keep a page out of search results, let crawlers fetch it and use a noindex robots meta tag or X-Robots-Tag header instead. A crawler that is blocked cannot read that noindex.
https://example.com/robots.txt does not apply to https://shop.example.com or http://example.com. The tester reads the file for the exact protocol and host of the URL you enter.
How this tool works, step by step
Read robots.txt once
The tester requests /robots.txt on the URL’s host with DeepSpider’s own user agent. It follows up to five redirects, even to another host, as RFC 9309 and Google describe. It never pretends to be Googlebot or another crawler. If you paste an edited file, nothing is requested and the pasted text is tested instead.
Interpret the response
A successful response is read as rules. Google and RFC 9309 treat a 4xx response (except 429) as if there were no robots.txt, so everything is allowed. A 429, a 5xx error or no answer means crawlers must assume everything is blocked for now. More than five redirects is treated as a missing file. A bot challenge leaves the result unknown.
Choose the group
Crawlers follow the group with the most specific User-agent line that matches their name, ignoring case. All groups that name the same crawler are combined. Only crawlers without a group of their own follow the * group. Googlebot-Image follows the Googlebot group when there is no Googlebot-Image group.
Find the deciding rule
Every Allow and Disallow rule in that group is compared with the URL path and query. * matches any characters and $ marks the end of the URL. When several rules match, the one with the longest path wins; when an Allow and a Disallow are equally long, Allow wins. The report shows the deciding line and every other rule that matched.
Check the file
The tester lists lines that crawlers ignore — unknown fields, rules before any User-agent line, retired noindex rules — and lines that Google accepts but other crawlers may not, such as a missing colon or a misspelled field. It also lists Sitemap and Crawl-delay lines and warns when the file is larger than the 500 KiB Google reads.
The live robots.txt for a host is reused for ten minutes, so testing several URLs or crawlers on the same website does not request the file again.
How to read the result
The crawler may fetch it
No rule in the crawler’s group blocks the URL, or an Allow rule wins. It does not mean the page is indexed or ranks; it only means robots.txt does not stand in the way.
A Disallow rule wins
The report names the line. If the block is not intended, change or remove that rule, or add a longer Allow rule for the path you want crawled.
robots.txt is unreachable
A server error, 429 or timeout makes Google pause crawling the whole host while it retries. Fix the server so robots.txt answers 200, or 404 if you do not want rules.
Challenged
A firewall challenged DeepSpider’s request for robots.txt. Real crawlers may be treated differently; your firewall logs show what they receive.
If your file has a group for Googlebot, Googlebot ignores every rule in the * group. Repeat the * rules you still want inside the Googlebot group. This is the most common reason a rule “does not work”.
Changing robots.txt safely
Paste your edited file and test the URLs that matter most: the homepage, key templates and the areas you want blocked. A broad Disallow can block far more than intended.
Blocking a URL in robots.txt does not remove it from search. Use noindex for pages that should not appear, and leave them crawlable so the noindex can be read.
Search engines render pages with their CSS and scripts. Blocking those files can stop them understanding how the page looks and works.
Several groups for the same crawler are combined, but they are easy to miss when editing. Keep each crawler’s rules together and repeat shared * rules where needed.
Sitemap lines must be full URLs. They apply to every crawler, wherever they appear in the file.
Answer robots.txt with 200, or 404 if you have no rules. Server errors and rate limits on robots.txt make Google stop crawling the whole host until it can read the file.
Google ignores Crawl-delay. Bing documents that Bingbot honours it, and some other crawlers do too, so a large Crawl-delay can slow those crawlers down a lot. Use it only if your server truly cannot handle the load.
Frequently asked questions
Does the tester request my page?
No. It requests only robots.txt (or nothing, when you paste an edited file). It does not check whether the page exists, is indexed or carries a noindex. Use the free Indexability Checker for those signals.
Why does the tester not use Googlebot’s user agent?
A request that only borrows Googlebot’s name comes from our servers, not Google’s. Many firewalls verify crawlers by IP address and would treat it as an impostor. The tester reads the file with DeepSpider’s own user agent and evaluates the rules as each crawler would.
Is a URL blocked by robots.txt removed from Google?
No. Google may still index a blocked URL if other pages link to it, usually without a snippet. Use noindex for pages you want out of search results, and keep them crawlable.
Why is my * rule ignored for Googlebot?
Because the file also has a group naming Googlebot. A crawler follows only its most specific group, so the * group no longer applies to it. Copy the rules you need into the Googlebot group.
How are Allow and Disallow conflicts decided?
The rule with the longest matching path wins. Google counts the whole pattern, so “Disallow: /*.htm” beats “Allow: /page” for /page.htm. When an Allow and a Disallow rule are equally long, Allow wins.
Do wildcards work?
Yes. Google, Bing and the AI crawlers support * for any characters and $ for the end of the URL. Rules are case-sensitive: “Disallow: /Fish” does not block /fish.
Why can other crawlers read my file differently?
Google’s open-source robots.txt parser also accepts a few misspellings, such as “Disalow”, and a field without its colon. RFC 9309, the robots.txt standard, defines the exact field names followed by a colon, so a crawler that follows it literally skips those lines. Bing and the AI crawlers do not document how they treat them. The tester lists such lines, says which crawlers are documented to read them and shows the result for your URL without them. Spell every field exactly (User-agent, Allow, Disallow, Sitemap) and add the colon so every crawler reads the same rules.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay. Bingbot honours it, and other crawlers vary.
Where can I see what Google actually read?
For a website you own, the robots.txt report in Google Search Console shows which robots.txt files Google found, when it last read them and any errors, and lets you ask Google to read the file again after a change.
Where to check the underlying guidance
Read How Google interprets the robots.txt specification, the standard RFC 9309, Google’s list of common crawlers, the robots.txt report in Search Console, Bing’s note on Crawl-delay and Bingbot and Yoast’s guide to robots.txt.