Crawl budget is the intersection of two things: the crawl rate limit (how much Google can fetch without straining your server) and crawl demand (how much Google wants to fetch, based on popularity and staleness).
Who actually needs to think about it
Google is explicit that most sites do not. It becomes relevant for sites with more than roughly a million URLs that change regularly, or mid-size sites (10,000+ URLs) with very frequently changing content. Below that, if pages are not being indexed, the cause is almost always quality or discoverability rather than budget.
What wastes it
- Faceted navigation generating endless URL combinations
- Session IDs and tracking parameters in URLs
- Soft 404s that return 200
- Infinite calendars and paginated archives with no end
- Duplicate content across parameter variations
Alien Road recommends: before optimising crawl budget, check the server log. We ask for raw logs on every large-site engagement because they show what Googlebot actually requested, and in most cases the crawl is being spent on parameter URLs nobody knew existed. Fixing that is faster and safer than any directive you can add.
How we apply this
Crawl budget matters on large sites and is a distraction on small ones. Our threshold matches Google’s: below roughly ten thousand URLs we do not touch it, and above that we start with server logs rather than with crawl-budget theories.
Related services