Robots.txtUsing robots.txt
robots.txt sets crawling parameters for search robots. Restricting sections reduces server load and speeds the site up. Yandex supports the…
Robots.txt
Robots.txtThe User-agent directive
The robot looks for records beginning with User-agent: and matches either the substring Yandex (case-insensitively) or *. The precedence rules…
Disallow and Allow directives
Disallow prevents crawling of sections or pages; Allow permits them. Yandex lists what belongs behind Disallow: pages with confidential data,…
Robots.txtThe Sitemap directive
The Sitemap directive declares where the sitemap lives. Several may be listed, one per line: User-agent: Yandex Allow: / sitemap:…
The Crawl-delay directive
Crawl-delay has been ignored by Yandex since 22 February 2018. Any site still carrying the directive is carrying a line…
Robots.txtThe Clean-param directive
Clean-param is a Yandex extension with no equivalent elsewhere: it tells the robot which URL parameters do not change the…
Robots.txtHow to check that a robot belongs to Yandex
Robots can claim to be Yandex by setting the user agent. Verification is done by reverse DNS lookup, and the…
Robots.txtHow do I lower a load on the site or server?
The robot plans its own visits and adjusts load automatically. Requests can spike when it discovers a new section, a…