The robot finds pages by following links. Yandex states the consequence without softening it: a document with no links pointing to it will never be discovered and will be ignored in search. Structure is therefore not an aesthetic matter.
The six rules
- Keep a clear link structure. Every document belongs to a section and must be reachable through a real
<a href>link. Yandex adds a fact worth planning around: indexing time depends partly on nesting depth — the deeper a page sits, the longer it may take to be included. Most visitors also enter at the home page, because a domain is easier to remember than an internal URL. - Use a sitemap on large projects to help the robot find and analyse documents.
- Restrict indexing of technical pages. Duplicates, internal search results, statistics pages — Yandex explains exactly why this matters: such pages are added and updated constantly, so they get crawled often while pages with important information go unnoticed. This is the crawl-budget argument stated in the engine’s own words.
- Give every page a unique, descriptive URL. Transliteration helps the robot infer the subject; the example given shows how much a well-formed URL communicates before the page is even fetched.
- Provide text links to sections, so the robot learns what they contain.
- Check symlinks so URLs cannot grow indefinitely. Pages with repeated tokens —
example.com/vasya/vasya/vasya/— may not be indexed at all. This is the classic infinite-space trap produced by relative links in a mishandled template.
The encoding detail
Use the same encoding for pages and for Cyrillic URLs. When the robot finds href="/корзина" on a UTF-8 page it stores the link in that encoding, so the page must be reachable at the percent-encoded equivalent. A mismatch here produces links that work in a browser and 404 for the robot.
What to do with this
Check three things on any site: that every page worth indexing is reachable by a plain link, that nothing important sits more than three or four levels deep, and that filter and search URLs are excluded. Those three cover most of what this page describes and most of what goes wrong.
So setzen wir das um
The rule we act on first is the third one, because Yandex explains the mechanism so clearly: technical pages change constantly, so they attract the crawler while the pages that matter wait. On large catalogues, blocking filter and internal-search URLs measurably speeds up how quickly new products appear. The infinite-URL symlink trap is rarer but brutal — we test for it by requesting a deliberately repeated path and checking the response is a 404 rather than a page.
Verwandte Leistungen