Your Crawl Budget Is Now Shared with Every Google Crawler

A mid-sized outdoor shop adds 400 new products in the spring. The pages are good, the sitemap is updated the same day and the products go into the Merchant Center feed. A week later, half of the new pages still haven't been crawled. The server logs are busy, though. Googlebot is there every day, and so are Storebot-Google, AdsBot-Google and Googlebot-Image. Thousands of requests go to sorted category pages, old redirects and a set of filter URLs nobody remembers creating.

For years the advice for a site like this was a little vague: "crawl budget is mostly a big-site problem, don't worry about it." In July 2026 Google rewrote its crawl budget documentation, and one sentence in the new version changes how you should read those logs:

"While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others."

So the shop isn't dealing with one crawler and one budget. It's dealing with one capacity limit for its hostname, and several Google crawlers that each want a share of it. This article explains what that means in practice, which sites should care, where crawling usually leaks away, and how to measure it on your own site, including what DeepSpider.ai can and can't show you.

In short
Google sets one crawl capacity limit per hostname, and every Google crawler draws on it: Googlebot, the image and video crawlers, Storebot-Google, AdsBot and the rest. Each crawler has its own demand, so a busy one can leave less room for the others. You can't buy more capacity, but you can stop wasting it: fewer duplicate and broken URLs, faster and more stable responses, and support for 304 Not Modified.

What Google actually changed in July 2026

Google updated its crawl budget guide, which now lives in its crawling infrastructure documentation, on 22 July 2026. Barry Schwartz and Glenn Gabe flagged it the same morning. Google said the edits were meant to "improve clarity, terminology consistency, and flow", and Schwartz read most of them as "not fundamental changes to how Google crawls". Several details are stated more precisely, though.

Is the sharing new? Probably not. In 2020 Schwartz asked John Mueller whether Google's search and ads crawlers share a budget. Mueller said the goal is to keep crawling from taking a server down, "so you have to count all the requests, regardless of the crawler type", and added: "It's not new". In March 2026 Gary Illyes described Googlebot as "just a user of something that resembles a centralized crawling platform", with Google Shopping, AdSense and dozens of other clients sending their requests through the same infrastructure. What has changed is that it's now in the guide, so you can plan around it.

One hostname, one capacity limit

The easiest way to think about crawl capacity is as connection time. Google keeps a number of connections open to your server, for a certain length of time. The limit isn't "5,000 pages a day". It's closer to "this much of your server's time". A page that answers in 150 milliseconds costs a fraction of a page that takes two seconds, so the same limit covers far more fast pages than slow ones.

These crawlers share the limit when they request URLs on the same hostname:

• Googlebot Smartphone and Googlebot Desktop, which crawl for Google Search.
• Googlebot-Image, Googlebot-Video and Googlebot-News, which fetch media and news content.
• Storebot-Google, which crawls for Google Shopping, and the merchant feed demand that comes with it.
• AdsBot-Google and Mediapartners-Google, which check landing pages and ad content.
• GoogleOther and its image and video variants, which Google uses for research and development. Google says they have "no effect on any specific product".
• Google-InspectionTool, which runs when you or someone else uses URL Inspection or the Rich Results Test.

Two names often cause confusion. Google-Extended isn't a crawler. It's a token you can use in robots.txt to say whether your content may be used for Gemini training and grounding. Google says it "doesn't have a separate HTTP request user agent string", so it never shows up in your logs and can't use any capacity. Blocking it also "does not impact a site's inclusion in Google Search".

The second confusion is about AI crawlers from other companies. You'll find articles claiming that GPTBot or ClaudeBot eat into your Google crawl budget. They don't, at least not directly. Google's limit covers Google's crawlers. But they do use the same server. If a burst of AI crawling slows your responses or triggers 429 or 5xx errors, Google sees an unhealthy site and lowers its limit. The effect is indirect, and it's real.

The volumes can be large. On Vercel's network in late 2024, GPTBot made 569 million requests in a month and Anthropic's crawler 370 million, against 4.5 billion for Googlebot. Wikimedia explained why such traffic costs more than its share: bots "bulk read" many pages, including less popular ones that are less likely to be cached, so they made up about 35% of its pageviews but at least 65% of its most expensive traffic. Cache misses hit your origin server, where slow responses start.

Subdomains and CDNs

Because a site is a hostname, www.example.com and images.example.com are crawled under separate limits. That's useful. If product images, PDFs or a large help centre live on their own hostname, they don't compete with your HTML pages for the same capacity. It also cuts the other way. A brand-new hostname starts at the same conservative default as any new site, so moving content to a fresh subdomain doesn't give it a head start.

Do you need to worry about this?

Most sites don't. Google's guide is aimed at three kinds of site, and it calls the numbers "a rough estimate" rather than "exact thresholds":

• Large sites, with over a million unique pages, whose content changes about once a week.
• Medium or larger sites, with over 10,000 unique pages, whose content changes daily.
• Sites where a large share of URLs sit in Search Console as Discovered - currently not indexed.

For a 300-page brochure site, Google will find and refresh everything without help, and you'll get more from improving the pages themselves. But shared capacity makes a few more sites worth a closer look than the size thresholds suggest. If any of these sound familiar, read on:

• You run a shop with a Merchant Center feed, and new or changed products take days to be crawled.
• You run dynamic ads, so AdsBot has a reason to visit often.
• You have a large image library on the same hostname as your pages.
• Your logs show many requests to parameter URLs, sorted lists or old redirects.
• Your server slows down at certain times of day, or returns 429 or 503 under load.

Patrick Stox makes a useful point in his Ahrefs guide: popular pages are rarely the problem. It's usually pages that are newer, rarely changed or poorly linked from the rest of the site that Google doesn't crawl often. Newer sites can feel it too, because Google may not yet want to crawl as much as the server could handle. The simple test still applies: if your important pages are crawled and updated as often as you need, you don't have a crawl budget problem. Shared capacity doesn't change that test. It changes where you look when the answer is no.

How the limit grows and shrinks

Google describes the capacity limit as something that moves. It starts at the same default for every site. It rises "over time" when there's demand and the site stays healthy, and it falls when the site struggles.

In the guide's words, the limit goes up when "the site responds consistently and its response times (including latency and Time-to-First Byte) remain stable or improve". It goes down when the site slows down or responds with server errors. Google's other crawling documentation adds that 500, 503 and 429 responses make its crawlers slow down. Gary Illyes put it simply in the March 2026 post: if your server struggles, "our crawlers will automatically back off to avoid overloading your infrastructure, which will drop your crawl frequency".

Two practical consequences follow.

Response time is part of your crawl budget. It isn't only a user experience number. A page that takes 1.5 seconds to generate holds a connection ten times longer than a cached page that answers in 150 milliseconds. Caching HTML at the edge, fixing slow database queries and keeping server-side rendering fast all give Google more room under the same limit.

Emergency throttling has a cost. Google's own advice for an emergency, when crawling is overloading your server, is to return 500, 503 or 429 for a short while, a day or two at most. It lists what that costs: fewer new pages discovered, slower refreshes of existing pages, removed pages staying in the index longer, and ads that may be paused because AdsBot can't check them. Don't return 403 or 404 to slow Google down. Google has warned against this explicitly, because those codes mean something else and can get pages dropped.

The Crawl-delay line doesn't help

Some sites add a Crawl-delay line to robots.txt to protect their servers. Googlebot ignores it. Bing says Bingbot honours it, and some other crawlers follow it too. If Google is crawling too hard, the fix is server capacity, caching or the short-term error codes above.

Demand: Google has to want the pages

Capacity is the ceiling. Demand decides how close Google gets to it. The guide is blunt about this: "Even if the crawl capacity limit isn't reached, if crawl demand is low, Google will crawl your site less."

Each crawler has its own demand. Google gives two examples: AdsBot "generally has a higher demand when a site is running dynamic ad targets", and Google Shopping "has a higher demand for products you have in your merchant feeds". For Googlebot, demand depends on "a site's size, update frequency, page quality, and relevance, compared to other sites". Google names three factors you can influence:

• Perceived inventory. Without guidance, Google tries to crawl most URLs it knows about. If many are duplicates or URLs you don't care about, that wastes crawling. Google calls this "the factor that you can positively control the most".
• Popularity. URLs that are more popular on the web tend to be crawled more often.
• Staleness. Google wants to recrawl pages often enough to notice changes. Site moves cause a temporary rise in demand while Google reprocesses the new URLs.

This is why the shop in the opening example isn't helped by a bigger server alone. If Google's demand for the new products is modest and most of the existing capacity goes to filter URLs, the new pages wait. The work is to make the useful URLs easy to find and the useless ones hard to reach.

Where crawl capacity leaks away

Every request Google spends on a URL you don't need is a request it doesn't spend on one you do. On most sites with a crawl problem, the same handful of patterns account for the waste.

Parameters, filters and sorting

Faceted navigation is the classic leak. A category with five filters and three sort orders can produce thousands of URL combinations, most of them near-duplicates. Google's faceted navigation guide lists the two problems this causes: overcrawling, and slower discovery of new pages because crawling time goes to the combinations.

Google's options, in order of how firmly they stop crawling:

• Block filter URLs in robots.txt if they don't need to be in search results. A rule such as Disallow: /*?*sort= stops Googlebot requesting sorted variants.
• Use URL fragments (#color=red) for filters that don't need their own URLs. Google generally doesn't treat fragments as separate URLs.
• Use rel=canonical to point variants at the main version. This consolidates signals, but Google still has to crawl a URL to see its canonical, so it saves less crawling than robots.txt.
• Add rel=nofollow to every filter link, consistently. It's a hint, so it's the weakest of the four.

If some filter pages should rank, such as "red trail shoes", give them clean, consistent URLs: the same parameter order every time, standard & separators, and a 404 for combinations that have no products.

Tracking parameters belong in the same bucket. A link from your own newsletter page to /shoes?utm_source=newsletter creates a new URL for Google to crawl. Session IDs in URLs are worse, because every visit can create a new one. Keep tracking on external campaign links, not on links between your own pages.

When to leave it alone
Not every parameter is a leak. Merchant Center auto-tagging adds srsltid to your links in Google's results. John Mueller has said it doesn't affect crawling, and when Riccardo Mares at Oncrawl analysed three months of logs from one large shop, he counted about 570 hits on srsltid URLs out of roughly 5.3 million. For crawling, keep canonical tags right, have your cache ignore it, and move on. Oncrawl does suggest switching auto-tagging off if you can, because srsltid URLs end up indexed alongside the clean ones and are hard to see in Search Console and GA4.

Soft 404s

A soft 404 is a page that says "not found" or shows an empty template while returning HTTP 200. Google keeps crawling these, because nothing in the response says the page is gone. The guide says plainly that soft 404 pages "will continue to be crawled, and waste your budget". The common causes are discontinued products that show an empty product template, search pages with no results, and sites that answer every unknown URL with the homepage.

Redirects and redirect chains

Each redirect is a request. A chain such as /sale → /sale/ → /offers/ costs three requests to reach one page, and Google says long chains "have a negative effect on crawling". The first hop here is a trailing slash mismatch. Old redirects keep getting requested for years if internal links, canonicals or sitemaps still point at them. Update the links to the final URL and redirect each old URL straight to its destination in one step.

Test redirects with a crawler, not only a browser. Glenn Gabe once found thousands of URLs that sometimes sent crawlers, Googlebot included, to the homepage with a 302, while the same URLs loaded normally when he checked them in a browser. The cause was a load balancing problem.

Removed pages: 404 and 410 beat robots.txt

When a page is permanently gone, return 404 or 410. Google says a 404 is "a strong signal not to crawl that URL again". It's tempting to block removed URLs in robots.txt instead, but blocked URLs "will stay part of your crawl queue much longer, and will be recrawled when the block is removed".

Noindex isn't a crawl saver

Noindex keeps a page out of the index, but Google has to request the page to see the tag. Google's guide is direct about it: "Don't use noindex, as Google will still request, but then drop the page". If you never want a URL crawled, use robots.txt. If you want it crawled but not indexed, noindex is right, and you accept the crawl cost.

There's one more trap here. Google says not to use robots.txt "to temporarily reallocate crawl budget for other pages". Blocking a section for a week doesn't move that capacity to your new products unless Google is already hitting your capacity limit. Block things you never want crawled, and leave it at that.

Sitemaps that tell the truth

Google reads your sitemaps regularly. Keep them to the URLs you want crawled and indexed: working, canonical, indexable pages. Add <lastmod> when content changes. Google uses lastmod only when it's "consistently and verifiably" accurate. If every URL shows today's date on every build, Google learns to ignore it. Google ignores the priority and changefreq fields entirely.

HTML that's too large

Googlebot fetches the first 2 MB of an HTML file, including the HTTP headers. The rest is ignored, without an error. This isn't a crawl budget issue in the usual sense, but it's the same infrastructure, and Gary Illyes covered it in the same post. Most pages are nowhere near 2 MB of HTML. The ones that get close usually carry inline base64 images, large blocks of inline CSS or JavaScript, or huge menus before the content. Keep important tags, structured data and the main content near the top. Note that 2 MB is the size of the HTML document, not the total weight of the page with images and scripts.

HTTP caching: the saving most sites never claim

If a page hasn't changed since Google last fetched it, your server can answer with 304 Not Modified and no body. Google reuses its stored copy. The guide recommends it directly, and Google's December 2024 post on HTTP caching explains why it's worth doing: only about 0.017% of Google's fetches were cacheable, down from about 0.026% ten years earlier. Almost nobody uses it.

Google supports both kinds of validator:

• ETag and If-None-Match. The server sends a version tag with the page. Next time, Googlebot sends it back, and the server answers 304 if the tag still matches. Google "strongly recommends" ETag, and if both validators are present, ETag wins.
• Last-Modified and If-Modified-Since. The same idea with a date. Use the standard HTTP date format, such as Fri, 4 Sep 1998 19:15:56 GMT.

You can check your own server in a minute. Request a page once, copy the ETag from the response, then send it back:

curl -sI https://www.example.com/shoes/trail-runner-2

# copy the ETag value, for example "v42"

curl -sI -H 'If-None-Match: "v42"' https://www.example.com/shoes/trail-runner-2

The second response should be 304 Not Modified. If you get 200 with the full page every time, your server or CDN isn't using conditional requests.

Two cautions. First, only change the ETag or Last-Modified date when the content changes in a way that matters. Google's own example is that a new copyright year in the footer isn't a significant change. If a rotating "related products" block changes the tag on every request, 304 never happens. Second, the opposite mistake is worse. A server that returns 304 when the page has changed tells Google to keep an outdated copy. That's the failure to test for after any caching change.

The other crawlers on your hostname

Once you know the limit is shared, look at who else is using it. A few questions help:

• Is Storebot-Google or AdsBot a large share of requests? That's expected if you run Shopping or dynamic ads, and you usually want it. Large spikes after a feed change are worth watching if they coincide with slower Googlebot crawling of new pages.
• Is Googlebot-Image fetching many images from the same hostname as your pages? Serving images from a separate hostname or CDN gives them their own limit.
• Is GoogleOther busy? It doesn't affect any specific Google product, according to Google, and it respects its own robots.txt token. If it uses a meaningful share of a stretched server, you can disallow it without affecting Search.
• Are non-Google bots slowing the server? If bursts from GPTBot, ClaudeBot or other bots coincide with slow responses or 429s, rate-limit them at the CDN or add robots.txt rules for the bots you don't want.

Don't block Googlebot-Image or Storebot-Google to "save budget" for Googlebot unless you're sure you don't need image search or Shopping. Those crawlers do real work for you. The aim is to stop waste, not to starve useful crawlers.

How to measure it on your site

There are two good sources, and they answer different questions.

Search Console's Crawl Stats report (Settings → Crawl stats) shows what Google saw from its side: total requests, download size, average response time, and host status for robots.txt fetching, DNS and server connectivity. It breaks requests down by response code, file type, purpose (discovery of new URLs or refresh of known ones) and Googlebot type, with separate rows for AdsBot and StoreBot, so it's the quickest place to see the shared limit at work. It's only available for root-level properties, and Google says sites with fewer than a thousand pages shouldn't need it. Glenn Gabe calls it "a (pretty) good proxy for log files", though when he wrote that in 2021, its data lagged three to four days behind.

Your server or CDN access logs show every request, from every crawler, against every URL. That's where you can see which crawlers share your hostname, which URLs they spend time on and whether new pages are being reached. The catch is that logs are large and messy, and anyone can fake a Googlebot user agent. Requests should be verified against the IP ranges Google publishes before you trust them. Logs are most useful next to a crawl of your own site: Screaming Frog's log guide matches the two to find orphan URLs that search engines still crawl, and linked URLs they never request.

What to look for, in order:

• The share of Googlebot requests that end in redirects, 404s, 5xx errors or parameter URLs.
• How long it takes for new or updated pages to receive their first Googlebot request.
• Whether average response time rises when other crawlers, including AI bots, are busy.
• Whether crawling drops after days with server errors or 429s, and how quickly it recovers.
• Which sections of the site Googlebot visits often, and which it rarely reaches. For a rarely reached section, check how many clicks it sits from the homepage.

Total crawl volume is left off the list on purpose. Jes Scholz calls the total number of crawls "nothing but a vanity metric" and tracks crawl efficacy instead: the delay between publishing or significantly updating a page and Googlebot's next visit, from your database and logs.

How DeepSpider.ai helps

DeepSpider looks at crawling from both sides: what your site serves when a crawler visits, and what Google's crawlers actually requested. You need both. An audit finds the URLs that could waste crawling, and logs show whether Google is in fact spending time on them. That's the context an audit issue count needs before you decide what to fix first.

Bot activity reads your access logs and shows what each crawler did. Your browser reads the log files on your computer. Each request that claims to be a known crawler is checked against the IP ranges that Google, Microsoft, OpenAI, Anthropic, Perplexity and Apple publish, and only a summary of crawler counts is uploaded. No IP addresses or visitor lines leave your computer. If you'd rather not upload files by hand, you can have your host or CDN send logs automatically. The overview separates Googlebot Smartphone and Desktop from the other Google crawlers, and from Bingbot and the AI crawlers, so you can see how the shared capacity on your hostname is actually used.

The reports that matter most for crawl budget:

• Googlebot requests on non-indexable URLs shows how many verified Googlebot requests went to redirects, 404s, other client errors, 5xx and 429 responses, parameter URLs, pages the audit found non-indexable, pages with a canonical to another URL, and exact duplicates.
• Indexable pages Googlebot did not request lists pages from your audit that Googlebot didn't visit in the period, with a filter for pages in your XML sitemap.
• Pages Googlebot requests that the audit did not find shows URLs Google knows about that your own links no longer lead to.
• Server errors served to Googlebot and Status codes over time show when crawl health suffered.
• Crawl activity by site section shows where Googlebot spends its time, and how much of each section's indexable pages it reached.
• AI crawler activity shows what GPTBot, ClaudeBot, PerplexityBot and others request, so you can see whether they compete for the same server.

The site audit finds the causes. Each leak in this article has a matching check:

If you connect Google Search Console, DeepSpider also checks the index status of your indexable audited pages through URL Inspection, so you can tell pages Google hasn't crawled from pages it crawled and chose not to index.

It's worth being clear about the limits too. DeepSpider doesn't read the Crawl Stats report, so it can't show Google's host status history or its own response time measurements. One audit crawl isn't Googlebot's view: DeepSpider crawls from its own servers in Europe, at its own pace. Slow HTML response is one measurement from that server, and Incorrect 304 responses tests a sample of pages now, not the responses Google received last month. And Bot activity can only show what's in the logs you give it. If your CDN serves most crawler requests from cache and you only upload origin logs, those requests are missing. Use the audit to find the causes and the logs to check whether Google is affected.

A 30-day plan

If crawling looks like a problem on your site, this order works well:

Week 1: measure

Download 30 days of access logs and upload them to Bot activity, or set up automatic delivery. Note the share of Googlebot requests that end in waste, and how the other Google crawlers compare. Check Crawl Stats in Search Console for average response time and host status problems. Run a full site audit.

Week 2: stop the biggest leak

Pick the largest waste category from the logs. It's usually parameter URLs or redirects. Remove tracking parameters from internal links, block sorted and filtered variants you don't need in robots.txt, and update links and canonicals that point to redirects. Fix soft 404s so removed pages return 404 or 410.

Week 3: make responses cheaper

Turn on ETag or Last-Modified support and check that unchanged pages return 304. Make sure validators only change when content changes. Look at the slowest templates, and cache HTML where you can. If other bots cause slowdowns, rate-limit them at the CDN.

Week 4: check the effect

Upload the new logs and compare. You're looking for a lower share of wasted requests, a shorter delay before new pages get their first Googlebot visit, and a stable or falling response time. Don't expect overnight change. Google adjusts capacity "over time", and some old URLs will keep being requested for months. Re-run the audit to confirm the fixes are live, then leave the robots.txt rules alone. Changing them every week sends mixed signals.

What success looks like
Before: 43% of Googlebot requests went to parameter URLs, redirects and missing pages, and new products waited a week for their first crawl. After: wasted requests are under 15%, unchanged pages return 304, and new products are requested within a day or two. Your numbers will be different, and that's fine. Compare your site with itself, not with someone else's.

Common myths about shared crawl budget

"AI crawlers use up my Google crawl budget." They don't use Google's limit. They do use your server, and if they slow it down, Google's limit falls.

"Blocking Google-Extended saves crawl budget." It can't. Google-Extended sends no requests. It only controls how Google may use your content for Gemini.

"Noindex saves crawl budget." It doesn't. Google has to fetch the page to see the tag.

"A bigger server means more crawling." It raises the ceiling. Google still needs demand for your pages to crawl more.

"Crawl budget only matters for million-page sites." The size guidance still holds, but a smaller shop with Shopping, dynamic ads and a lot of filter URLs can feel the shared limit much earlier.

Frequently asked questions

Is crawl budget a ranking factor?

No. Crawling decides whether and when Google sees a page. It doesn't make a page rank better. But a page Google hasn't crawled can't be indexed or updated in search results.

Can I ask Google for more crawl budget?

Not directly. Google lists two ways to get more: add server resources if the site can't be crawled because of server capacity, for example when URL Inspection reports "Hostload exceeded", and improve the quality of your content for the Google product you're targeting.

Do images count against my page crawling?

If they're on the same hostname, the requests from Googlebot-Image share the same capacity limit. Images on a separate hostname or CDN have their own.

Should I block Storebot-Google or AdsBot to give Googlebot more room?

Only if you don't use Shopping or Google Ads. Blocking them stops the products that depend on them. Fix waste first.

Does Crawl-delay in robots.txt work for Google?

No. Googlebot ignores it. Bingbot and some other crawlers follow it.

How long does it take to see a change?

It varies. Waste falls as soon as Google stops finding the bad URLs, but Google keeps recrawling URLs it already knows for a while, and capacity grows gradually. Compare logs a month apart rather than day to day.

The short version

Google gives each hostname one crawl capacity limit, and every Google crawler draws on it. Each crawler brings its own demand, so a busy Shopping or ads crawler can leave Googlebot less room. You can't buy more capacity, but you can stop wasting what you have: clean up parameters, redirects, soft 404s and sitemaps, answer quickly and consistently, and support 304 for pages that haven't changed. Then check your logs to see whether Google noticed.

Sources