The DeepSpider crawler,
and how it visits websites.

What DeepSpider does

DeepSpider.ai is a cloud service for technical SEO audits. A customer with a DeepSpider.ai account enters a website address and starts an audit. DeepSpider then reads the website the way a search engine crawler does and reports technical problems, such as broken links, redirect chains, missing titles or pages that robots.txt blocks.

Every audit is started by a customer. DeepSpider does not crawl websites on its own initiative, does not build a search index and does not collect content to train AI models. What it reads is used for the audit report of the customer who started the audit.

Customers can audit any public website. DeepSpider does not check who owns the website, but our Terms of Service allow customers to audit only websites they own, manage or are otherwise authorised to audit, and to respect robots.txt and the website’s access controls. Several customers can audit the same website independently.

Other automated requests

A few smaller features also send requests to websites. Each is started by a person or serves a website a customer is auditing:

User agents

DeepSpider names itself in the User-Agent header of every request. These are the exact values:

DeepSpider/1.1 SEO audit
DeepSpider/1.0 SEO audit
DeepSpider/1.0 (+https://deepspider.ai/)
DeepSpider/1.1 Daily SEO Brief

The optional mobile comparison, and the smartphone option of the free Redirect Chain Checker, send a smartphone user agent that also names DeepSpider:

Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 (compatible; DeepSpider/1.1 SEO audit; mobile comparison)

To match every DeepSpider request in a firewall or log filter, match the text DeepSpider/ in the user agent.

How DeepSpider reads robots.txt

During an audit DeepSpider reads robots.txt before it requests pages, images or sitemaps from a host, and follows it for every one of them. It follows the rules as Google documents them and as RFC 9309 describes:

• Its token is DeepSpider. Case does not matter, and a version such as DeepSpider/1.1 is read as DeepSpider. Other names, such as DeepSpiderBot or spider, do not apply to DeepSpider.
• One group applies. If robots.txt has groups for User-agent: DeepSpider, DeepSpider follows those groups only. Otherwise it follows the User-agent: * groups.
• The most specific rule wins. The longest matching Allow or Disallow path decides; on a tie Allow wins. * and $ work as wildcards.
• Every host has its own file. Images and links on other hosts follow that host’s robots.txt.
• Changes apply quickly. During an audit DeepSpider keeps robots.txt for at most 5 minutes before reading it again.
• When robots.txt cannot be read. A 4xx answer other than 429 means there are no rules, as RFC 9309 says. After HTTP 429, a server error or no answer, DeepSpider requests no pages from that host until it can read robots.txt.

Crawl-delay

DeepSpider follows Crawl-delay in the group that applies to it. It waits that many seconds between requests to the host and, for a delay of one second or more, sends one request at a time. The wait is shared by all DeepSpider audits of that host running at the same time.

DeepSpider waits at most 60 seconds. If robots.txt asks for longer, DeepSpider ends the audit instead of crawling more slowly, and the remaining URLs are not requested. On another host, such as an image server, a longer delay leaves its URLs unchecked.

Speed, rate limits and pauses

The customer chooses one of two crawl speeds:

• Super Dynamic (the default) starts with two requests at a time. It speeds up only after a run of quick, successful answers and slows down as soon as answers get slower or errors appear.
• Standard requests one page at a time.

One audit never sends more than 8 requests at the same time to one host. When the website answers in ways that ask for less traffic, DeepSpider reacts as follows:

When a paused audit continues

An audit paused by the website continues by itself after 30 minutes, or after 1 or 2 hours when the previous continuation checked nothing new. It stops continuing by itself after three continuations in a row without progress, after 30 continuations, or 7 days after it paused. The customer can also continue it.

Each continuation is slower than the last. The kind of request the website refused, pages or images, is sent one at a time with a gap of 1 second after the first pause, then 2, 4 and 8 seconds.

Only the audited website’s own answers pause or end an audit. A slow or refusing image host or linked website leaves its URLs unchecked and does not stop the audit.

What DeepSpider requests

• Pages on the audited host, with and without www, found through links, sitemaps and redirects, up to the customer’s plan limit: 5,000, 100,000 or 250,000 HTML pages per audit, or more on a custom package. A redirect to another domain is recorded, not followed.
• robots.txt and XML sitemaps: the sitemaps named in robots.txt and common locations such as /sitemap.xml.
• Images used on the pages, including images on other hosts, to check their status, file size and dimensions.
• CSS and JavaScript files, a limited number per audit, to check that they load. They are read as text and never run.
• Technical checks: the HTTP and HTTPS versions of the start address, the www and non-www version, and one made-up address such as /deepspider-missing-page-check-3f9a1c2b7d4e to see how the website answers a missing page. These use HEAD or GET.
• Links to other websites: at most 150 per audit, one request each, without reading the page. DeepSpider does not crawl other websites’ pages.
• Optional comparisons on the homepage and up to 100 other pages: the same page requested again with cache or language headers. They are skipped when the website is behind Cloudflare, has a Crawl-delay or has recently asked DeepSpider to slow down.

DeepSpider sends only GET and HEAD requests and no cookies. It reads up to 5 MB of each HTML page. It does not sign in, submit forms or run JavaScript, so pages behind a login cannot be audited.

Identifying DeepSpider

• User agent. Every request names DeepSpider (see User agents). DeepSpider never pretends to be another crawler or leaves its name out of the user agent, and never sends requests through proxies or other networks to get past a block.
• IP addresses. DeepSpider has no fixed IP addresses. Requests come directly from our hosting provider’s servers in Frankfurt, Germany, and the addresses can change.
• Web Bot Auth. DeepSpider has published a signed Web Bot Auth key directory at https://app.deepspider.ai/.well-known/http-message-signatures-directory and has applied to become a Cloudflare verified bot. Signing of crawl requests will be switched on after Cloudflare verifies DeepSpider; until then requests are not signed.

Anyone can send the DeepSpider user agent. If a request claims to be DeepSpider and you are unsure, send us the details listed under Contact.

Limit or block DeepSpider

Block DeepSpider completely:

User-agent: DeepSpider
Disallow: /

Keep DeepSpider out of some paths. A DeepSpider group replaces your User-agent: * rules for DeepSpider, so repeat in it every rule you want DeepSpider to follow:

User-agent: DeepSpider
Disallow: /checkout/
Disallow: /search/

Slow DeepSpider down by adding Crawl-delay to the group DeepSpider follows. At 5 seconds DeepSpider sends at most 720 requests an hour. A value above 60 ends audits of your website.

User-agent: DeepSpider
Crawl-delay: 5
Disallow: /checkout/

You can test your rules with our free robots.txt Tester.

Firewalls. A rule that matches DeepSpider/ in the user agent stops every DeepSpider request. When a website refuses DeepSpider, the audit pauses or ends and the customer sees the website’s answer. To let DeepSpider in instead, read Allow DeepSpider through your firewall.

Contact and support

For questions or problems with DeepSpider on your website, email support@deepspider.ai or use our contact form. Please include:

• the website (domain) and the URLs DeepSpider requested,
• the date and time of the requests, with the time zone,
• the user agent and, if you have it, the IP address you saw,
• what happened, for example the number of requests or the HTTP status codes.

DeepSpider.ai is operated by Netzforscher OÜ (registry code 17625017), Väike-Kuke tn 8b-1, 80018 Pärnu, Pärnu County, Estonia. How we handle personal data is described in our Privacy Policy.