The Crawl-delay directive

Crawl-delay has been ignored by Yandex since 22 February 2018. Any site still carrying the directive is carrying a line that does nothing.

What replaced it

The crawl rate setting in Yandex Webmaster, under Indexing → Crawl rate. It is automatic by default, calculated to load as many pages as possible without overloading the server, and can be set manually per site — including separately for every subdomain.

Why the change is an improvement

A file directive is static and blind: it applies the same delay whether the server is idle or struggling, and it slows the robot even when there is capacity to spare. The console setting is a recommendation Yandex weighs against what it knows about the site, and the automatic mode adapts.

Before reaching for it at all

Yandex’s guidance on crawl rate applies here: find out what the robot is actually requesting before slowing it down. If the load comes from filter combinations, session URLs and internal search results — as it usually does — the correct remedy is Disallow or Clean-param, which removes the wasted requests and returns the crawl budget to real pages. Reducing the rate keeps the waste and slows everything else down with it.

Housekeeping

Remove Crawl-delay from robots.txt when you next edit it. It has no effect on Yandex, some other crawlers still honour it, and leaving it in place suggests a rate control exists where none does — which is exactly the kind of assumption that outlives the person who wrote the file.

Where a genuine limit still lives

If robot traffic is a real operational problem, the console setting is the only Yandex-facing control, and it must be configured for the primary domain and each subdomain separately. A rate set on the main domain does nothing for the subdomain that is actually under load.