Using robots.txt

robots.txt sets crawling parameters for search robots. Restricting sections reduces server load and speeds the site up. Yandex supports the Robots Exclusion Protocol with its own extensions.

The warning that comes first, and matters most

Pages restricted in robots.txt can still appear in Yandex search. To remove a page from results you must use the noindex directive in the HTML or the HTTP header — and you must not block that page in robots.txt, because a blocked page is never fetched and the noindex is never seen.

This is the most consequential misunderstanding in technical SEO, and Yandex states it in the first paragraph. Blocking is about crawl budget; noindex is about search presence. Using the first to achieve the second guarantees the page stays indexed permanently.

File requirements

• Maximum 500 KB.
• A TXT file named robots.txt.
• In the root directory.
• Accessible to robots, returning 200 OK.

If the file does not meet these requirements, the site is considered open for indexing. A malformed or unreachable file is not a safety net — it is an open door.

The redirect feature

Yandex accepts a redirect from one site’s robots.txt to another site’s, and applies the directives in the target file. That is specifically useful during a site move, where both domains must present consistent rules — and it is why the migration guidance insists the two files agree.

The supported directives

• User-agent — which robot the rules apply to.
• Disallow — prohibit crawling of sections or pages.
• Sitemap — the path to the sitemap, given in full with protocol.
• Clean-param — a Yandex extension declaring parameters to ignore.

Verification

Test the file in the console’s robots.txt analyser with actual URLs, not just for syntax. A file that parses cleanly and blocks the wrong directory reports no errors at all.