Site indexing

Yandex’s indexing guidance is a checklist with specific technical limits attached — the limits are the part worth memorising.

The setup steps

1. Add the site in Yandex Webmaster.
2. Submit a sitemap, which can also express crawl priority — telling the robot which pages update frequently so it plans accordingly.
3. Configure robots.txt: Disallow for technical and low-value sections, and Clean-param for insignificant CGI parameters. Yandex spells out the failure mode for session identifiers: formally distinct URLs with identical content, and if there are many, the robot indexes those instead of downloading useful content.

The hard limits

• Documents over 10 MB are not indexed.
• Many CGI parameters, deeply nested directories, or excessively long URLs can prevent indexing.
• PDFs: only text is indexed. Text rendered as an image is not — which makes every scanned brochure and image-based catalogue invisible.
• Open Office XML and OpenDocument formats are indexed, though support for new formats can lag.
• .swf files are indexed where directly linked or embedded via object o embed; content inside a Flash file can lead the robot back to the containing HTML document.
• frameset and frame content is indexed, with the source document identified from the frame contents.

Two server-side requirements

• Non-existent URLs must return 404. On receiving it, the search engine removes the document from the index — which is the mechanism, and the reason a soft-404 leaves deleted pages in place indefinitely. Everything that should be indexed must return 200 OK.
• HTTP headers must be correct, and Yandex singles out the response to If-Modified-Since: the Last-Modified header must carry the document’s genuine modification date. A server that lies about this — returning the current time for every page — forces re-downloads of unchanged content and spends crawl capacity for nothing.

The practical priorities

Of everything here, two items produce most of the benefit on real sites: correct 404 handling, and honest Last-Modified with working conditional responses. Both are server configuration, both are invisible in a browser, and both are usually wrong.