Robots.txtUsing robots.txt
robots.txt sets crawling parameters for search robots. Restricting sections reduces server load and speeds the site up. Yandex supports the…
Robots.txt
Robots.txtThe User-agent directive
The robot looks for records beginning with User-agent: and matches either the substring Yandex (case-insensitively) or *. The precedence rules…
Directivas Disallow y Allow
Disallow impide el rastreo de secciones o páginas; Allow lo permite. Yandex enumera lo que debe ir detrás de Disallow:…
Robots.txtThe Sitemap directive
The Sitemap directive declares where the sitemap lives. Several may be listed, one per line: User-agent: Yandex Allow: / sitemap:…
La directiva Crawl-delay
Crawl-delay ha sido ignorado por Yandex desde el 22 de febrero de 2018. Cualquier sitio que aún mantenga esta directiva…
Robots.txtThe Clean-param directive
Clean-param is a Yandex extension with no equivalent elsewhere: it tells the robot which URL parameters do not change the…
Robots.txtHow to check that a robot belongs to Yandex
Robots can claim to be Yandex by setting the user agent. Verification is done by reverse DNS lookup, and the…
Robots.txtHow do I lower a load on the site or server?
The robot plans its own visits and adjusts load automatically. Requests can spike when it discovers a new section, a…