Friday, February 17, 2023<\/p>\n
\n Over the last few months we noticed an uptick in website owners and some content delivery networks \n The short version of this blog post is: please don’t do that; we have documentation about \n The \n The one exception is \n Client errors are just that: client errors. They generally don’t suggest an error with the server: \n And again, the big exception is the \n Alle \n We have extensive documentation about \n If you need more tips or clarifications, catch us on
\n (CDNs) attempting to use 404<\/code> and other 4xx<\/code> client errors (but not
\n 429<\/code>) to attempt to reduce Googlebot’s crawl rate.\n<\/p>\n
\n how to reduce Googlebot’s crawl rate<\/a>.
\n Read that instead and learn how to effectively manage Googlebot’s crawl rate.\n<\/p>\nBack to basics:
4xx<\/code> errors are for client errors<\/h2>\n4xx<\/code> errors servers return to clients are a signal from the server that the
\n client’s request was wrong in some sense. Most of the errors in this category are pretty benign:
\n “not found” errors, “forbidden”, “I’m a teapot” (yes, that’s a thing). They don’t suggest anything
\n wrong going on with the server itself.\n<\/p>\n429<\/code>, which stands for “too many requests”. This error is a clear
\n signal to any well-behaved robot, including our beloved Googlebot, that it needs to slow down
\n because it’s overloading the server.\n<\/p>\nWhy
4xx<\/code> errors are bad for rate limiting Googlebot (except 429<\/code>)<\/h2>\n
\n not that it’s overloaded, not that it’s encountered a critical error and is unable to respond
\n to the request. They simply mean that the client’s request was bad in some way. There’s no
\n sensible way to equate for example a 404<\/code> error to the server being overloaded.
\n Imagine if that was the case: you get an influx of 404<\/code> errors from your friend accidentally
\n linking to the wrong pages on your site, and in turn Googlebot slows down with crawling. That
\n would be pretty bad. Same goes for 403<\/code>, 410<\/code>, 418<\/code>.\n<\/p>\n429<\/code> status code, which translates to “too many
\n requests”.\n<\/p>\nWhat rate limiting with
4xx<\/code> does to Googlebot<\/h2>\n4xx<\/code> HTTP status codes (again, except 429<\/code>) will cause your content
\n to be removed from Google Search. What’s worse, if you also serve your robots.txt file with a
\n 4xx<\/code> HTTP status code, it will be treated as if it didn’t exist. If you had a rule
\n there that disallowed crawling your dirty laundry, now Googlebot also knows about it; not great
\n for either party involved.\n<\/p>\nHow to reduce Googlebot’s crawl rate, the right way<\/h2>\n
\n how to reduce Googlebot’s crawl rate<\/a>
\n and also about
\n how Googlebot (and Search indexing) handles the different HTTP status codes<\/a>;
\n be sure to check them out. In short, you want to do either of these things:\n<\/p>\n\n
500<\/code>, 503<\/code>, or 429<\/code> HTTP status code to Googlebot
\n when it’s crawling too fast.\n <\/li>\n<\/ul>\n
\n Twitter<\/a> or post in
\n our help forums<\/a>.\n<\/p>\n