How Search Works

Crawl Budget: When It Matters and When It Does Not

Crawl budget is the intersection of two things: the crawl rate limit (how much Google can fetch without straining your server) and crawl demand (how much Google wants to fetch, based on popularity and staleness).

Who actually needs to think about it

Google is explicit that most sites do not. It becomes relevant for sites with more than roughly a million URLs that change regularly, or mid-size sites (10,000+ URLs) with very frequently changing content. Below that, if pages are not being indexed, the cause is almost always quality or discoverability rather than budget.

What wastes it

  • Faceted navigation generating endless URL combinations
  • Session IDs and tracking parameters in URLs
  • Soft 404s that return 200
  • Infinite calendars and paginated archives with no end
  • Duplicate content across parameter variations

Alien Road recommends: before optimising crawl budget, check the server log. We ask for raw logs on every large-site engagement because they show what Googlebot actually requested, and in most cases the crawl is being spent on parameter URLs nobody knew existed. Fixing that is faster and safer than any directive you can add.

, ,
Teilen

© Copyright 2026 Alien Road. All rights reserved.