Crawl-delay: The robots.txt Line That Slows Bing and Does Nothing for Google

Open a few hundred robots.txt files and you will keep meeting the same line: Crawl-delay: 10. Sometimes it is 5, sometimes 30, and once in a while someone has typed 120. It usually sits under User-agent: *, it has been there for years, and nobody on the current team remembers who added it or why.

Every guide about it says the same three things. Google ignores it. Bing and Yandex follow it. If you want to slow Google down, use Search Console. Two of those three are no longer true, and the guides skip the more interesting question: if Google ignores the line, who is actually paying for it?

In short
Crawl-delay does nothing for Googlebot and, since 2018, nothing for Yandex. It still slows Bingbot, AI crawlers such as ClaudeBot and most SEO tools, and in Bing it even overrides the crawl settings you chose in Bing Webmaster Tools. Unless a real crawler is hurting your server today, delete the line. If one is, there are better tools for that.

The advice everybody repeats is out of date

Crawl-delay was never part of the robots.txt standard. The Robots Exclusion Protocol, RFC 9309, published in 2022, defines user-agent, allow and disallow; everything else is an extension that each crawler may or may not read. That is why the answer to "does Crawl-delay work?" has always been "for whom?".

Google. Its robots.txt documentation lists the fields it supports and adds, in brackets, that other fields such as crawl-delay aren’t supported. Nothing new there. What is new is the second half of the usual advice. Google retired the crawl rate setting in Search Console on 8 January 2024, saying its crawler now adjusts by itself and the tool was rarely needed. So "use Search Console instead" now leads to a page that no longer exists.

Yandex. Many guides, including some updated this year, still say Yandex follows Crawl-delay. According to Yandex’s own documentation, as quoted by PPC Land, it stopped on 22 February 2018. Crawl speed for Yandex is set in Yandex Webmaster.

Bing. Bing still reads the line, and reads it in a way most people don’t expect. The Bing Webmaster blog explains that the value is not a pause between requests but the size of a time window, from 1 to 30 seconds, in which Bingbot crawls your site at most once. With Crawl-delay: 5 that is around 17,280 pages a day at most. With 10 it is 8,640, which is also the figure Conductor uses.

Who actually pays for the line

If the biggest search engine ignores the line, it is tempting to think it is harmless. It isn’t. It just sends the bill somewhere else.

• Bing. Fewer pages per day means older copies of your pages in Bing. Bing says so itself: crawling fewer pages may mean less content indexed, which is why it usually does not recommend a delay. Bing’s index also feeds other search products, so the slowdown reaches further than bing.com.
• AI crawlers. Anthropic says ClaudeBot and its other crawlers support Crawl-delay. OpenAI’s crawler page does not mention it either way, so don’t assume. If you want AI assistants to see your current pages, a delay written for search engines ten years ago is now working against that.
• SEO tools. AhrefsBot follows the delay strictly for HTML pages. SemrushBot waits up to 10 seconds and cuts longer values down to 10. Site audit tools that you or your agency run follow it too, which is why an audit of a site with Crawl-delay: 10 can take a whole day.
• You. The irony is that the line rarely protects the server from what actually loads it. Aggressive scrapers don’t read robots.txt at all.

The numbers grow quickly. For a crawler that follows the delay exactly, one pass over a 20,000-page site takes a little over two days at 10 seconds, almost a week at 30 and close to a month at 120.

Three things most guides don’t mention

The line overrides your Bing Webmaster Tools settings

Bing Webmaster Tools has a Crawl Control setting where you can tell Bingbot to crawl harder at night and gently during your busy hours. It is the tool most site owners actually want. But Bing’s own 2012 blog post says a crawl delay in robots.txt overrides what you set there. If you ever wondered why the hourly chart in Bing Webmaster Tools seemed to change nothing, check robots.txt first.

Each subdomain has its own delay

Robots.txt applies to one host. Bing handles the robots.txt of each subdomain separately, so www, shop and blog on the same server each get their own delay. A delay of 1 on three subdomains can still mean three requests a second to one machine. If you really are protecting one small server, the line in one file doesn’t do what you think it does.

A named group replaces your general rules

A common "fix" is to move the delay into a group for one crawler, such as User-agent: bingbot. That is fine, but remember how robots.txt groups work: a crawler that finds a group with its own name follows only that group and ignores the User-agent: * group. If your Disallow lines for /cart/ or /admin/ live under *, copy them into the named group too, or that crawler will start fetching pages you meant to keep out.

So should you delete it?

For most websites, yes. Before you do, spend five minutes on two questions.

• Where does the line come from? On many sites robots.txt is generated by the CMS, an SEO plugin or the hosting panel. Editing the file on the server may be undone at the next update; change it where it is made.
• Is a crawler really causing load today? Look at your server logs or your host’s statistics, not at the robots.txt history. If nothing is hammering the server, the line is protecting you from nothing.

If a real crawler is overloading the server, there are better tools than a permanent delay for everyone:

• For an emergency, answer 503 or 429. Google’s guidance on reducing crawl rate is to return 500, 503 or 429 with a Retry-After header for a short time, a couple of hours or one to two days, not longer, because URLs that keep failing for days can drop out of the index.
• For Bing, use Crawl Control instead of the robots.txt line, so you can choose the hours.
• For one misbehaving bot, target that bot. Give it its own group with its own delay or Disallow, and copy your general rules into it.
• For a slow server, fix the server. Caching and a CDN help every visitor, crawlers included. A delay only hides the symptom for the crawlers polite enough to read it.

How DeepSpider.ai handles Crawl-delay

DeepSpider is one of the crawlers that read the line, and we would rather be honest about what that means for your audit. Up to 60 seconds, DeepSpider follows it exactly: one request at a time, with the requested gap between them, and the audit shows that it is waiting at the website’s request. The time it spends waiting is listed separately, so you can see how much of the audit duration the line cost.

Longer than 60 seconds, an audit would take days, so DeepSpider ends it with a clear explanation instead of crawling for a week. It links the robots.txt file, shows the exact line, works out what the delay would mean for 1,000 pages and lists the steps to fix it.

A delay on someone else’s website, such as a page you link to, only slows the requests to that website. It never pauses the audit of your own site.

Frequently asked questions

Does Crawl-delay affect my Google rankings?

Not directly. Googlebot ignores the line, so it changes nothing about how Google crawls you. The effects show up in Bing, in AI assistants that use their own crawlers and in the SEO tools you rely on.

Is Crawl-delay: 1 harmless?

Mostly. One request per second is gentle for any server built in the last decade, and Bingbot can still fetch tens of thousands of pages a day. If the line serves no purpose, though, removing it is still the cleaner choice.

Can I comment the line out instead of deleting it?

Yes. Put # in front of it. Crawlers ignore comments, and the next person to read the file can see that the delay was removed on purpose.

The short version

Crawl-delay is a line written for a web where servers were small and Google listened. Today Google doesn’t, Yandex doesn’t, and the crawlers that still do are exactly the ones you probably want to see your site: Bing, AI assistants and your own audit tools. Find out where the line comes from, check whether anything really loads your server, and if nothing does, delete it.

Sources