Sitemap files

A sitemap tells Yandex the site’s current structure. It can be declared in robots.txt or added on the Sitemap files page; either way, the robot downloads it within two weeks, and every file — including each file listed in an index file — is processed separately.

The four statuses

• OK — parsed and loaded; indexed pages appear in results within two weeks.
• Redirect — the sitemap URL redirects. Remove the redirect.
• Error — malformed; the link shows details.
• Not indexed — the server answered with something other than 200, or robots.txt blocks access to the file. Check with the server response tool using the full path.

Updating: a quota worth knowing

An updated file does not need to be removed and re-added — the robot checks regularly. Processing can be forced with the refresh control, and each file in an index file can be triggered individually. But this is limited to 10 uses per host, after which you wait 30 days from the first attempt, with the exact date shown in the console. Spending all ten in a single afternoon of impatience leaves nothing for the migration two weeks later.

Strategy for large sites

Yandex’s own advice is counter-intuitive and correct: the sitemap does not have to contain everything. Pages Yandex already knows can be dropped, leaving new and frequently updated pages — and with an index file, the volatile files marked with lastmod. Crawl statistics shows what Yandex already knows. This turns the sitemap from a static inventory into a change feed, which is what the robot actually uses it for.

Two confusing behaviours, explained

• The sitemap appears as an excluded page with “invalid document format”. Informational only. The robot tried to index the XML as an ordinary page; it does not affect processing and the file does not participate in search.
• “Unknown tag” error. The file contains elements Yandex does not support — a reference to a mobile version, for example. Unsupported elements are ignored and supported ones processed, so the file does not need changing.

Removal

Delete the file in the console and remove the Sitemap directive from robots.txt, or the robot keeps finding it. It clears from the database within a few weeks.