BLOG CATEGORY

Crawling

Crawl discovery, scale, limits, errors and crawler behaviour.

Technical SEO · Crawling

Crawl-delay: The robots.txt Line That Slows Bing and Does Nothing for Google

Google ignores Crawl-delay, Yandex stopped reading it in 2018 and Search Console no longer has a crawl rate setting. So who still pays for the line? Bing, AI crawlers and your own SEO tools.

Read the guide →

Crawling · Technical SEO

Your Crawl Budget Is Now Shared with Every Google Crawler

Google now says it plainly: one crawl capacity limit per hostname, shared by Googlebot, Shopping, ads and image crawlers. Learn what that means, where crawling leaks away, and how to measure it in your own logs.

Read the guide →