{"id":28512,"date":"2026-09-08T21:56:08","date_gmt":"2026-09-08T18:56:08","guid":{"rendered":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/the-crawl-delay-directive\/"},"modified":"2026-09-09T00:47:43","modified_gmt":"2026-09-08T21:47:43","slug":"the-crawl-delay-directive","status":"publish","type":"ar_ykb","link":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/the-crawl-delay-directive\/","title":{"rendered":"The Crawl-delay directive"},"content":{"rendered":"<p><strong><code>Crawl-delay<\/code> has been ignored by Yandex since 22 February 2018.<\/strong> Any site still carrying the directive is carrying a line that does nothing.<\/p>\n<h2>What replaced it<\/h2>\n<p>The <strong>crawl rate setting<\/strong> in Yandex Webmaster, under Indexing \u2192 Crawl rate. It is automatic by default, calculated to load as many pages as possible without overloading the server, and can be set manually per site \u2014 <em>including separately for every subdomain<\/em>.<\/p>\n<h2>Why the change is an improvement<\/h2>\n<p>A file directive is static and blind: it applies the same delay whether the server is idle or struggling, and it slows the robot even when there is capacity to spare. The console setting is a recommendation Yandex weighs against what it knows about the site, and the automatic mode adapts.<\/p>\n<h2>Before reaching for it at all<\/h2>\n<p>Yandex&#8217;s guidance on crawl rate applies here: find out what the robot is actually requesting before slowing it down. If the load comes from filter combinations, session URLs and internal search results \u2014 as it usually does \u2014 the correct remedy is <code>Disallow<\/code> ou <code>Clean-param<\/code>, which removes the wasted requests and returns the crawl budget to real pages. Reducing the rate keeps the waste and slows everything else down with it.<\/p>\n<h2>Housekeeping<\/h2>\n<p>Remove <code>Crawl-delay<\/code> from <code>robots.txt<\/code> when you next edit it. It has no effect on Yandex, some other crawlers still honour it, and leaving it in place suggests a rate control exists where none does \u2014 which is exactly the kind of assumption that outlives the person who wrote the file.<\/p>\n<h2>Where a genuine limit still lives<\/h2>\n<p>If robot traffic is a real operational problem, the console setting is the only Yandex-facing control, and it must be configured for the primary domain and each subdomain separately. A rate set on the main domain does nothing for the subdomain that is actually under load.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Crawl-delay has been ignored by Yandex since 22 February 2018. Any site still carrying the directive is carrying a line that does nothing. What replaced it The crawl rate setting in Yandex Webmaster, under Indexing \u2192 Crawl rate. It is automatic by default, calculated to load as many pages as possible without overloading the server, [&hellip;]<\/p>\n","protected":false},"menu_order":88,"template":"","meta":{"footnotes":""},"ar_ykb_kategori":[765],"ar_ykb_etiket":[],"class_list":["post-28512","ar_ykb","type-ar_ykb","status-publish","has-post-thumbnail","hentry","ar_ykb_kategori-robots-txt"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28512","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_ykb"}],"version-history":[{"count":2,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28512\/revisions"}],"predecessor-version":[{"id":29119,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28512\/revisions\/29119"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/28894"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=28512"}],"wp:term":[{"taxonomy":"ar_ykb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_kategori?post=28512"},{"taxonomy":"ar_ykb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_etiket?post=28512"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}