The crawl-delay directive is a command within a website’s robots.txt file that instructs search engine crawlers, specifically Yandexbot, to wait a designated number of seconds between successive requests. By implementing this directive, webmasters manage server resources more effectively, preventing the bot from overwhelming the hosting infrastructure during intensive indexing periods. This setting helps maintain site speed for human visitors while ensuring search engines still access content at a sustainable pace.
Why should you use the crawl-delay directive for Yandex?
Managing how often a search spider visits your site is a key part of technical maintenance. When a server receives hundreds of requests per minute from a spider, it must allocate CPU cycles and memory to handle each process. For sites with complex backends or large databases, this resource allocation can lead to slower response times for actual customers. Yandexbot can be quite active, and if your server is not configured to handle high concurrency, you might see a spike in 503 Service Unavailable errors.
A high crawl frequency is generally good for getting new content indexed quickly. However, the stability of the site is more important. If the server crashes or slows down significantly, the user experience suffers, which can indirectly affect your rankings. By using a delay, you provide a buffer that allows the server to breathe between bot hits. This is especially useful for sites running on shared hosting or VPS plans with limited resources.
We often see servers struggling with simultaneous requests from multiple bots. Setting a specific delay for Yandexbot prevents these spikes without stopping the indexing process entirely. It is a simple but effective way to ensure stability.
Alien Road SEO team
How to format the crawl-delay directive in robots.txt
The syntax for this directive is straightforward, but it must be placed in the correct section of your robots.txt file to be recognized. Yandex looks for this instruction specifically under the User-agent: Yandex or User-agent: * blocks. The value represents the number of seconds the bot should wait before starting a new request after the previous one has finished.
- Place the directive on its own line within the robots.txt file.
- Use a colon followed by a space and then the numerical value.
- Ensure the value is an integer or a decimal if the engine supports it.
- The directive should be part of a specific User-agent block for better control.
- Verify that there are no conflicting Allow or Disallow rules that might confuse the parser.
For example, if you want Yandexbot to wait 5 seconds between requests, your file would look like this: User-agent: Yandex followed by Crawl-delay: 5 on the next line. This tells the spider that it can only make one request every five seconds. If your site has 10,000 pages, it will take longer to crawl the entire site, but the load on your server will remain consistently low.
Does Yandex support fractional values for delays?
While many webmasters use whole numbers like 1, 2, or 10, Yandex documentation indicates that fractional values are also acceptable. For instance, a value of 0.5 would mean a half-second delay, allowing for two requests per second. This is a more aggressive setting than a full second but still provides a small gap that can prevent a literal flood of requests in a single millisecond.
It is worth testing different values to find the right balance. A delay that is too long might mean that your newest blog posts or product updates take weeks to appear in search results. Conversely, a delay that is too short might not solve your server performance issues. Most medium-sized websites find that a value between 0.5 and 2.0 works well for their needs.
How Alien Road applies the crawl-delay directive
In a typical engagement, our team starts by reviewing the server logs and the crawl statistics provided in search consoles. We look for patterns where bot activity correlates with high memory usage or increased latency for users. Alien Road uses a data-driven approach to determine if a delay is necessary or if the server simply needs better caching.
- Step 1: Analyze server logs to identify the peak frequency of Yandexbot requests.
- Step 2: Compare crawl times with server response times (TTFB) to see if the bot is slowing down the site.
- Step 3: Implement a conservative Crawl-delay value, such as 2, in the robots.txt file.
- Step 4: Monitor the Crawl Statistics report in Yandex Webmaster to ensure the bot is still visiting essential pages.
- Step 5: Adjust the value based on the speed of indexing and server health over a 14-day period.
For example, a Shopify store selling luxury goods might have thousands of filter combinations that Yandexbot tries to index. This can create a massive load. By applying a specific delay, we ensure the site remains fast for shoppers while still allowing Yandex to discover the most important product pages. We prioritize the user experience because a slow site eventually loses both customers and search rankings.
Can you use Yandex Webmaster tools instead?
Yes, Yandex provides a graphical interface within Yandex Webmaster to set the crawl rate. This is found under the ‘Indexing Settings’ menu. The tool allows you to move a slider to indicate the desired speed. This method is often easier for those who are not comfortable editing text files on their server. However, the robots.txt directive is considered more definitive by many technical SEOs because it resides directly on the root of the site.
If you set both a directive in robots.txt and a value in the webmaster panel, Yandex generally follows the instruction in the panel as it is a more recent and direct setting. We recommend using one method consistently to avoid confusion. If you have a developer who manages the server, the robots.txt file is usually the better place for this configuration.
While the directive is helpful, it is a blunt tool. We prefer combining it with server-side caching to reduce the actual processing weight of each bot visit. This way, even if the bot visits frequently, the server doesn’t have to work as hard.
Alien Road SEO team
What are the common mistakes with this directive?
One major error is setting a delay that is far too high, such as 60 seconds. A 60-second delay means the bot can only crawl 1,440 pages per day. For a large e-commerce site with 500,000 pages, it would take over a year to crawl the entire site once. This effectively kills your SEO performance. Another mistake is forgetting that different bots have different names. Using ‘User-agent: Googlebot’ with a crawl-delay won’t affect Yandex at all.
Syntax errors are also common. Placing the directive outside of a User-agent block makes it invalid. Similarly, using the wrong capitalization or spelling—like ‘Crawl-speed’ instead of ‘Crawl-delay’—will cause Yandexbot to ignore the rule entirely. Always use a validator tool to check your robots.txt file after making changes.
Where to start this week
Begin by checking your Yandex Webmaster account to see if there are any warnings about server availability. If you see frequent 5xx errors during crawl times, it is a clear sign you need to implement a delay. Add a simple Crawl-delay: 2 to your robots.txt file under the Yandex agent. Monitor your server performance for three days to see if the CPU load stabilizes. If you don’t see an improvement, you may need to look into deeper server-side optimizations or database indexing.