{"id":25670,"date":"2026-09-06T00:47:44","date_gmt":"2026-09-05T21:47:44","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/reduce-the-google-crawl-rate\/"},"modified":"2026-09-06T00:51:43","modified_gmt":"2026-09-05T21:51:43","slug":"reduce-the-google-crawl-rate","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/reduce-the-google-crawl-rate\/","title":{"rendered":"Reduce the Google crawl rate"},"content":{"rendered":"<p>\n  Google&#8217;s crawler infrastructure has sophisticated algorithms to determine the optimal crawl rate<br \/>\n  for a site. Our goal is to crawl as many pages from your site as we can on each visit without<br \/>\n  overwhelming your server. In some cases, Google&#8217;s crawling of your site might be causing a<br \/>\n  critical load on your infrastructure, or cause unwanted costs during an outage. To alleviate this,<br \/>\n  you may decide to reduce the number of requests made by Google&#8217;s crawlers.\n<\/p>\n<h2 id=\"understand-the-cause-of-the-sharp-increase-in-crawling\" tabindex=\"-1\">Understand the cause of the sharp increase in crawling<\/h2>\n<p>\n  Sharp increase in crawling may be caused by inefficiencies in your site&#8217;s structure or issues with<br \/>\n  your site otherwise. Based on the reports we&#8217;ve received in the past, the most common causes are:\n<\/p>\n<ul>\n<li>\n    Inefficient configuration of URLs on the site, which is typically casued by a specific<br \/>\n    functionality of the site:<\/p>\n<ul>\n<li>Faceted navigation or other sorting and filtering functionality of the site<\/li>\n<li>A calendar with a lot of URLs for specific dates<\/li>\n<\/ul>\n<\/li>\n<li>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/troubleshoot-google-search-crawling-errors\/#adsbot\">A Dynamic Search Ad target<\/a>\n  <\/li>\n<\/ul>\n<p>\n  We strongly recommend that you check with your hosting company and look at recent access logs of<br \/>\n  your server to understand the source of the traffic, and see if it fits in the aformentioned<br \/>\n  common causes of the sharp increase in crawling. Then, check our guides about<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/managing-crawling-of-faceted-navigation-urls\/\">managing crawling of faceted navigation URLs<\/a><br \/>\n  and<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/troubleshoot-google-search-crawling-errors\/#improve_crawl_efficiency\">optimizing crawling efficiency<\/a>.\n<\/p>\n<h2 id=\"urgently-reduce-crawler-traffic-for-emergencies\" tabindex=\"-1\">Urgently reduce crawler traffic (for emergencies)<\/h2>\n<aside class=\"warning\">\n  <b>Warning<\/b>: When considering reducing the Google&#8217;s crawl rate, keep in mind that this will<br \/>\n  have broad effects. For Search, Googlebot will discover fewer new pages, and existing pages will<br \/>\n  be refreshed less frequently (for example, prices and product availability may take longer to be<br \/>\n  reflected in Search), and removed pages may stay in the index longer. For Google Ads, your<br \/>\n  campaigns may be cancelled or paused, and your ads may not serve.<br \/>\n<\/aside>\n<p>\n  If you need to urgently reduce the crawl rate for short period of time (for example, a couple<br \/>\n  of hours, or 1-2 days), then return <code>500<\/code>, <code>503<\/code>, or <code>429<\/code> HTTP<br \/>\n  response status code instead of <code>200<\/code> to the crawl requests. Google&#8217;s crawling<br \/>\n  infrastructure reduces your site&#8217;s crawling rate when it encounters a significant number of URLs<br \/>\n  with <code>500<\/code>, <code>503<\/code>, or <code>429<\/code> HTTP response status codes (for<br \/>\n  example, if you<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/temporarily-pause-or-disable-a-website\/\">disabled your website<\/a>).<br \/>\n  The reduced crawl rate affects the whole hostname of your site (for example,<br \/>\n  <code>subdomain.example.com<\/code>), both the crawling of the URLs that return errors, as well as<br \/>\n  the URLs that return content. Once the number of these errors is reduced, the crawl rate will<br \/>\n  automatically start increasing again.\n<\/p>\n<aside class=\"warning\">\n  <b>Warning<\/b>: We don&#8217;t recommend that you do this for a long period of time (meaning, longer<br \/>\n  than 1-2 days) as it may have a negative effect on how your site appears in Google products. For<br \/>\n  example, in case of Search, if Googlebot observes these status codes on the same URL for multiple<br \/>\n  days, the URL may be dropped from Google&#8217;s index.<br \/>\n<\/aside>\n<h2 id=\"exceptional-requests-to-reduce-crawl-rate\" tabindex=\"-1\">Exceptional requests to reduce crawl rate<\/h2>\n<p>\n  If serving errors to Google&#8217;s crawlers is not feasible on your infrastructure,<br \/>\n  <a href=\"https:\/\/search.google.com\/search-console\/googlebot-report\" class=\"external-link\">file a special request<\/a><br \/>\n  to report a problem with unusually high crawl rate, mentioning the optimal rate for your site in<br \/>\n  your request. You cannot request an increase in crawl rate, and it may take several days for the<br \/>\n  request to be evaluated and fulfilled.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If Google&#8217;s crawl rate overwhelms your server, explore this document to learn how to reduce Google&#8217;s crawl rate and stop bots from crawling your site.<\/p>\n","protected":false},"menu_order":7000,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[708],"ar_kb_etiket":[],"class_list":["post-25670","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-crawling-infrastructure-crawling-and-indexing"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25670","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25670\/revisions"}],"predecessor-version":[{"id":25692,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25670\/revisions\/25692"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27665"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25670"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25670"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25670"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}