The User-agent directive

The robot looks for records beginning with User-agent: and matches either the substring Yandex (case-insensitively) or *. The precedence rules are strict and produce most robots.txt mistakes.

The precedence rules

• If User-agent: Yandex is present, User-agent: * is ignored entirely — not merged, ignored. Rules you assumed were global do not apply.
• If a block exists for a specific Yandex robot, such as YandexBot, then both the general Yandex block and * are unused for that robot.
• If neither Yandex nor * is found, the robot has unlimited access.

Yandex’s own example makes the trap concrete: a file with YandexBot, Yandex and * blocks applies only the first to the main indexing robot, only the second to other Yandex robots, and the third to neither.

Formatting

A blank line should precede every User-agent directive. # starts a comment, and everything to the end of the line is ignored — which, combined with the rule that an empty line after User-agent detaches the rules that follow, accounts for most silently broken files.

The exception nobody expects

Some Yandex robots may ignore robots.txt directives altogether, including those under User-agent: Yandex. To constrain those, you must name their exact user agent in its own block. This is why a site that believes it has blocked everything can still see crawling from a Yandex agent it has never heard of.

Practical shape of a file

Write one block for Yandex containing every rule that should apply to Yandex, and a separate complete block for *. Do not rely on inheritance between them, because there is none. Duplication here is correct, not redundant.

Verification

Test specific URLs per user agent in the analyser. The tool applies the same precedence rules the robot does, which is the only reliable way to confirm which block actually governs a given address.