Why missing pages and host variants matter
Missing pages. When someone requests an address that does not exist, the server should answer 404 Not Found or 410 Gone, ideally with a helpful page. If it answers 200, every mistyped or removed address looks like a working page. Google calls a page that says “not found” but answers 200 a soft 404, excludes such pages from search and may still spend crawling on them. Google’s John Mueller has said that redirecting missing pages to the homepage is treated as a soft 404 too.
Host variants. http://example.com, https://example.com, http://www.example.com and https://www.example.com are four different addresses. If more than one answers with the site, every page exists several times and links and signals are split. Google recommends choosing one and redirecting the others to it. The same applies to a page with and without a trailing slash: Google treats them as two URLs.
This checker shows which status the server returned. Whether Google classifies a page as a soft 404 also depends on what the page shows, which this check does not read. Use Search Console’s Page indexing report for Google’s own classification.
How this tool works, step by step
Find the main address
The checker requests the entered site’s homepage over HTTPS and follows its redirects within the site. Where it ends is the main address the other variants are compared with.
Request each variant
It requests the homepage over http and https, with and without www, and follows up to 6 redirects for each. A redirect to another site is reported but not followed.
Request an address that cannot exist
It requests one random address, /deepspider-missing-page-check- followed by 12 random characters, which no site links to, and records its answer and any redirects.
Test the trailing slash
If you enter a page such as example.com/blog, it requests the page with and without a trailing slash.
Use the audit’s rules
Requests use HEAD (a one-byte GET when HEAD is not allowed), DeepSpider’s own user agent and robots.txt rules for DeepSpider, exactly like the technical step of a DeepSpider audit. The audit’s own checks then judge the answers.
How to read the report
One permanent redirect
The variant reaches the main address with a single 301 or 308. A missing page answers 404 or 410.
Works, but not cleanly
A temporary 302 or 307, a chain of redirects, or both trailing-slash forms answering 200.
Duplicate or soft 404
A variant that answers 200 itself, ends on another address or fails, or a missing page that answers 200 or redirects to a working page.
Does not resolve
The www (or non-www) host has no DNS. It is not a duplicate; add DNS and a redirect only if people use that address.
A missing page that redirects and then answers 404 is shown for review: crawlers still see the error, but the redirect adds a request. A 401 or 403 means a firewall or the server refused the request, so the not-found handling could not be seen.
How to fix what the checker finds
Configure the server, CMS or JavaScript framework to answer 404 (or 410 for content removed on purpose) while still showing a helpful not-found page.
Redirect only old URLs that have a real replacement. Redirecting every unknown address to the homepage is treated like a soft 404 and confuses visitors.
Pick www or non-www and HTTPS, and redirect every URL on the other variants to the same path on the chosen one.
Send each variant straight to the final address with 301 or 308, for example http://example.com → https://www.example.com in one step instead of two.
Redirect one form to the other with a 301, and use the chosen form in links, canonicals and sitemaps.
Use the main address in canonical tags, hreflang, XML sitemaps and internal links so all signals point the same way as the redirects.
Frequently asked questions
Is a soft 404 a ranking penalty?
Google does not describe it as a penalty. Soft 404 pages are excluded from search results, and a site that answers 200 for every address can waste crawling on pages that do not exist.
Should I use 404 or 410?
Both tell Google the page does not exist. 410 says the removal is permanent; 404 is the general answer. Either is fine for unknown addresses.
Can I redirect deleted pages to the homepage?
Only when the homepage really replaces them. Google has said that redirecting many removed pages to the homepage is treated as a soft 404. Redirect to a closely matching page or answer 404 or 410.
Does it matter whether I choose www or non-www?
No. What matters is choosing one and redirecting the other to it, so the site has one address.
Why does the checker use HEAD requests?
HEAD returns the status and headers without the page, which is all this check needs. A DeepSpider audit uses the same requests. Some servers answer HEAD differently from GET; if a result looks wrong, check the address in a browser’s developer tools.
Why can my browser show a different result?
Servers and CDNs can answer differently by location, cookies or user agent, and some firewalls challenge automated requests. The report shows what DeepSpider’s server in the EU received with DeepSpider’s own user agent.
Why was something not checked?
DeepSpider honours robots.txt. If robots.txt disallows an address for DeepSpider, or cannot be read because of a server error, that address is not requested and the report says so.
Where to check the underlying guidance
Read Google’s Troubleshoot crawling errors (soft 404), HTTP status codes, Consolidate duplicate URLs, Redirects and Google Search and To slash or not to slash; Search Engine Roundtable on redirects to the homepage seen as soft 404s; Glenn Gabe’s case study on redirects to less relevant pages; and Sitebulb on URLs that resolve under both HTTP and HTTPS.