{"id":28469,"date":"2026-09-08T21:56:08","date_gmt":"2026-09-08T18:56:08","guid":{"rendered":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/crawl-rate\/"},"modified":"2026-09-09T00:47:43","modified_gmt":"2026-09-08T21:47:43","slug":"crawl-rate","status":"publish","type":"ar_ykb","link":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/crawl-rate\/","title":{"rendered":"Crawl rate"},"content":{"rendered":"<p>Crawl rate is the number of requests per second the robot makes to your site. It is set automatically by default, and Yandex&#8217;s algorithm aims to load as many pages as possible without overloading the server. It must be configured <strong>separately for the primary domain and each subdomain<\/strong>.<\/p>\n<h2>Before reducing it<\/h2>\n<p>Yandex asks for a diagnosis first, and the order is deliberate. If robot traffic is hurting the server, find out <em>what<\/em> the robot is requesting:<\/p>\n<ul>\n<li>Read the server logs.<\/li>\n<li>Open Indexing \u2192 Crawl statistics with the <em>All pages<\/em> view and look for service pages and duplicates \u2014 typically URLs with GET parameters.<\/li>\n<\/ul>\n<p>If the robot is spending its budget on filter combinations and session URLs, the correct fix is <code>Disallow<\/code> in <code>robots.txt<\/code>, verified with the robots.txt analysis tool \u2014 not a lower crawl rate. Reducing the rate treats the symptom and slows down the pages you actually want indexed. This distinction is the whole content of the page.<\/p>\n<h2>Increasing it<\/h2>\n<p>The value is a <em>recommendation<\/em>. Yandex says plainly that the actual rate may differ, based on what it knows about the site. A larger number does not buy faster indexing on a site the robot has no reason to crawl harder.<\/p>\n<h2>How to set it<\/h2>\n<p>Indexing \u2192 Crawl rate, switch on <strong>Set manually<\/strong>, move the slider \u2014 it starts at the calculated optimum \u2014 and save.<\/p>\n<h2>When manual is justified<\/h2>\n<p>Shared hosting where robot traffic measurably raises response times, a scheduled event where headroom matters, or a site behind a rate-limiting layer that starts returning errors under crawl load. Outside those cases, automatic is the better setting, and a manual value set years ago and forgotten is a quiet cap on how quickly new pages get discovered.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Crawl rate is the number of requests per second the robot makes to your site. It is set automatically by default, and Yandex&#8217;s algorithm aims to load as many pages as possible without overloading the server. It must be configured separately for the primary domain and each subdomain. Before reducing it Yandex asks for a [&hellip;]<\/p>\n","protected":false},"menu_order":45,"template":"","meta":{"footnotes":""},"ar_ykb_kategori":[758],"ar_ykb_etiket":[],"class_list":["post-28469","ar_ykb","type-ar_ykb","status-publish","has-post-thumbnail","hentry","ar_ykb_kategori-indexing"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28469","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_ykb"}],"version-history":[{"count":2,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28469\/revisions"}],"predecessor-version":[{"id":29060,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28469\/revisions\/29060"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/28851"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=28469"}],"wp:term":[{"taxonomy":"ar_ykb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_kategori?post=28469"},{"taxonomy":"ar_ykb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_etiket?post=28469"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}