How do I prohibit indexing of the site or pages?

Three instruments hide content from indexing, and they operate at different levels: the whole site or a section, an individual page, or part of a page’s text.

Site, section or page

• Disallow in robots.txt.
• The robots meta tag with noindex or none.
• Authorisation — and Yandex specifically recommends this for hiding a home page, with a reason worth remembering: a home page blocked in robots.txt can still appear in results if other sites link to it. Blocking is not concealment.

For anything genuinely restricted, combine authorisation with a server response of 404, 403 or 410 for unauthorised requests, so the page does not merely refuse to render but reports itself as unavailable.

Part of a page’s text

The noindex element — a Yandex extension, distinct from the meta tag:

<noindex>text that should not be indexed</noindex>

Two rules make it dangerous if misused:

• If the closing tag is missing, the entire remaining content of the page is treated as hidden. One unclosed tag can remove a whole page’s text from the index.
• Nested tags are ignored after the first closing tag, so nesting does not work as written.

For valid markup, the comment form is accepted: <!--noindex-->…<!--/noindex-->.

Choosing between them

Ask what should happen if someone types the URL directly. If the answer is “they should see it, but it should not rank” — noindex. If it is “the robot should not waste time here” — Disallow. If it is “nobody should see this” — authorisation, with an appropriate status code. The three are not interchangeable, and the most common mistake is using the second for the third.