Disallow and Allow directives

Disallow prevents crawling of sections or pages; Allow permits them. Yandex lists what belongs behind Disallow: pages with confidential data, internal search results, traffic statistics, duplicates, logs, and database service pages.

The recommendation people miss

For pages whose addresses carry GET parameters, Yandex recommends Clean-param rather than Disallow, and explains why: with Disallow, it may be unable to identify the duplicate’s parameter-free counterpart, and some of the blocked pages’ accumulated metrics are lost. Blocking discards signals; Clean-param consolidates them onto the canonical address.

This is a genuinely different answer from the one most technical audits give, and it applies to the largest category of unwanted URLs on any commercial site — tracking tags, sort orders, session identifiers.

Patterns

Disallow: /            # the whole site
Disallow: /catalogue   # anything beginning /catalogue
Disallow: /page?       # anything whose URL contains parameters after /page

Allow in combination

User-agent: Yandex
Allow: /cgi-bin
Disallow: /

Everything blocked except paths beginning /cgi-bin. This inversion — block all, permit specific — is the right shape for a staging environment or an application where only a few sections should ever be crawled, and it is far safer than enumerating every directory to exclude.

What Disallow does not do

It does not remove a page from search. A blocked page can still be listed. Worse, blocking a page prevents the robot from reading a noindex on it, so the two instruments must never be combined on the same URL.

Before deploying

Run the intended URLs through the analyser — both those that should be blocked and those that must not be. A rule of the wrong precision is invisible until traffic disappears.