Crawler Management

Diagnosing Crawl Errors

Not every crawl error is a problem. A 404 for a page you deliberately deleted is correct behaviour. The skill is telling deliberate from accidental.

Errors that need action

  • Server error (5xx), the page exists but the server failed. Sustained 5xx leads to removal from the index.
  • Redirect error, chains that are too long, loops, or redirects to a 404.
  • Blocked by robots.txt on a page you want indexed.
  • Soft 404, a “not found” message returned with status 200.
  • Submitted URL marked noindex, the sitemap and the page contradict each other.

Statuses that are usually fine

“Crawled, currently not indexed” often means Google read the page and judged it not worth storing; that is a content question, not a technical one. “Discovered, currently not indexed” usually means crawl priority, common on large new sites. “Alternate page with proper canonical tag” is the system working correctly.

Alien Road recommends: fix 5xx and redirect errors first, always. They are unambiguous and they compound: a redirect chain slows every crawl of that path, and a server error can quietly remove a page that ranked well for years.

, ,
Share

© Copyright 2026 Alien Road. All rights reserved.