Why an audit paused
or ended early.

When the website asks DeepSpider to slow down

Websites protect themselves from too many requests. When a server answers HTTP 429 (too many requests) or HTTP 503 (service unavailable) again and again, sends a Retry-After header, or stops answering for 10 minutes, DeepSpider pauses the audit. Everything checked so far is kept.

The audit shows Audit paused · DeepSpider continues in … with a countdown. You don’t need to do anything:

• DeepSpider continues 30 minutes after a pause, or at the time the website named in Retry-After.
• Each continuation goes slower, with 1, 2, 4 and then 8 seconds between requests, so the website has time to recover.
• If a continuation checks nothing new, the next wait is longer: 1 hour, then 2 hours. After three continuations without progress, DeepSpider stops trying and you decide.

A paused audit can be continued for 7 days. Stop cancels the automatic continuation; you can still press Continue saved audit later.

When the pause is on our side

Sometimes a very large audit needs more memory or a save takes longer than expected. Then the box says the pause is on DeepSpider’s side and that there is nothing to do on yours. DeepSpider continues after 2, 10 and 30 minutes, then after 2 hours, and can move a very large audit to a server with more memory. Continuing a paused audit never uses another monthly audit.

When the audit ends instead

Some situations will not get better by waiting, so the audit ends and the worker is freed for other audits. The box names the reason and, where it applies, what the website owner can change.

Finished, but with limits

A yellow result means the audit finished and names what limited it, for example Page limit reached, Sitemap blocked (HTTP 403) or images that were temporarily unavailable. The findings are valid for everything that was checked. View audit notes shows the details and, when the website can fix something, how.

If the limit was your own page or image limit, raise it in Settings and start a new audit. A finished audit cannot be extended.

Let DeepSpider in

If you own the website, these steps solve most pauses and blocks. Allow only what the audit needs, and remove the rule afterwards if you prefer.

1. Find what refused the requests. The audit box shows the HTTP status and, for Cloudflare, the Ray ID. Search for it in your firewall, CDN or hosting logs.
2. Allow the user agent DeepSpider/1.1 SEO audit. In Cloudflare, create a custom rule where User Agent contains DeepSpider/1.1 SEO audit with the action Skip. On the Free plan, Bot Fight Mode cannot be skipped and has to be turned off during the audit.
3. Check robots.txt. Remove a long Crawl-delay line, or comment it out with #. Googlebot ignores this line anyway.
4. Slow the next audit down if the server is small: Settings → General → Audit setup → Crawl speed → Standard.

DeepSpider runs from servers in Frankfurt, Germany. A website that only refuses visitors from outside its own country can block the audit even though it works for you. A user agent can be imitated, so keep an allow rule limited to the audit.

Does it use one of my monthly audits?

Not when the audit gave you nothing useful. An audit is not counted when the website blocked DeepSpider or left most pages unreachable before half of its page URLs were checked, when no HTML page could be audited, or when a problem on our side ended it. Continuing a paused audit is never a new audit. The full rule is in the Terms of Service.