The robot plans its own visits and adjusts load automatically. Requests can spike when it discovers a new section, a structural change, or a batch of new URLs. Yandex’s guidance is to diagnose before throttling.
The diagnosis
- Compare pages loaded with pages in search on the My sites page. A large gap means the robot is finding many similar or duplicate pages that should not be indexed.
- Open Crawl statistics, or the server logs, and look at what it actually fetched. Pay attention to URLs with GET parameters — filter combinations such as
?price=1,000&max_price=1,000,000, or internal search URLs such as?search=…. Yandex notes the waste plainly: the robot crawls these distinct URLs, and their identical content cannot participate in search anyway.
The two remedies, in order
Clean-param— directs the robot to the canonical URL among the duplicates, transfers accumulated metrics to it, and lets the load fall gradually.Disallow— the fast option: the robot stops accessing the specified pages within 24 hours of the file changing. Use it when the load is a live problem.
If the load is genuinely not about extra pages
Two options: configure the server to answer 429 Too Many Requests, which tells Yandex the server is struggling and it will reduce requests; or set the crawl rate manually in the console. The 429 route is preferable because it is dynamic — it applies while the server is under pressure and lifts when it is not.
Subdomains
Yandex treats each subdomain as a separate site and crawls each one, so the aggregate load can spike without any single site behaving unusually. If server logs show most requests going to subdomains, Yandex suggests something more radical than throttling: reconsider whether the subdomains are needed at all, and whether regional sections of the main site would serve visitors just as well.
Comment nous l'appliquons
The 24-hour figure for Disallow taking effect is the one to know during an incident — it is the fastest lever available. But we always look at the loaded-versus-in-search gap first, because the load is nearly always filter URLs, and blocking those returns both the capacity and the crawl budget. Configuring 429 is worth doing once on any site with a history of load problems: it is self-regulating in a way a fixed crawl rate is not.
Services associés