Crawl rate is the number of requests per second the robot makes to your site. It is set automatically by default, and Yandex’s algorithm aims to load as many pages as possible without overloading the server. It must be configured separately for the primary domain and each subdomain.
Before reducing it
Yandex asks for a diagnosis first, and the order is deliberate. If robot traffic is hurting the server, find out what the robot is requesting:
- Read the server logs.
- Open Indexing → Crawl statistics with the All pages view and look for service pages and duplicates — typically URLs with GET parameters.
If the robot is spending its budget on filter combinations and session URLs, the correct fix is Disallow in robots.txt, verified with the robots.txt analysis tool — not a lower crawl rate. Reducing the rate treats the symptom and slows down the pages you actually want indexed. This distinction is the whole content of the page.
Increasing it
The value is a recommendation. Yandex says plainly that the actual rate may differ, based on what it knows about the site. A larger number does not buy faster indexing on a site the robot has no reason to crawl harder.
How to set it
Indexing → Crawl rate, switch on Set manually, move the slider — it starts at the calculated optimum — and save.
When manual is justified
Shared hosting where robot traffic measurably raises response times, a scheduled event where headroom matters, or a site behind a rate-limiting layer that starts returning errors under crawl load. Outside those cases, automatic is the better setting, and a manual value set years ago and forgotten is a quiet cap on how quickly new pages get discovered.
Biz bunu nasıl uyguluyoruz
The first thing we check when a client asks us to slow the robot down is what it is actually fetching, because it is nearly always filter and parameter URLs rather than real pages. Blocking those returns both the server load and the crawl budget; lowering the rate returns only the load, and costs indexing speed on the pages that matter. We also make a point of looking for a manual rate someone set during a long-forgotten incident.
İlgili hizmetler