Site indexingHow does Yandex search work?
Yandex Search runs the same three stages as any modern engine — discovery, indexing, ranking — but the failure modes…
Site indexing
Site indexingHow can I add a site to search?
Pages appear in search after Yandex robots visit them. The question of how to get a site into search reduces…
What files can be included in search results?
Yandex indexes HTML plus a defined list of file formats, each up to 10 MB: PDF; Microsoft Office DOC, DOCX,…
Site indexingHow many site pages are included in search?
Counting how many of your pages are in Yandex is a two-source exercise: a site: query gives an approximation, the…
Site indexingHow do I check if a site is accessible to users and…
Availability is the precondition for everything else. Yandex makes the consequence explicit: if the robot cannot reach the site because…
Site indexingHow do I check if all important pages are included in search?
Aggregate indexing numbers hide the failure of individual pages that matter. Checking important pages explicitly is a separate discipline. How…
Site indexingHow do I inform Yandex about changes on the site?
Yandex’s recommendation for notifying it about new and updated pages is not to choose one method but to combine several:…
How to monitor and manage indexing?
Monitoring indexing means answering five questions, and the console has a report for each: when the robot came and what…
Site indexingHow do my site pages appear to indexing bots and users?
This page answers a question worth asking directly: how does a page look to an indexing bot, and what can…
Site indexingHow do I prevent from indexing the pages that aren’t needed in…
Most sites contain pages nobody should find in search: service and administration pages, content useful only to logged-in users such…
Site indexingWhy should I avoid duplicate pages?
Duplicate pages split signals and waste crawl budget, and on large catalogues they are generated automatically rather than written deliberately.…
Site indexingMoving the site
A site move in Yandex is a defined operation: group the old and new addresses, then designate the new one…
Robots.txtUsing robots.txt
robots.txt sets crawling parameters for search robots. Restricting sections reduces server load and speeds the site up. Yandex supports the…
Robots.txtThe User-agent directive
The robot looks for records beginning with User-agent: and matches either the substring Yandex (case-insensitively) or *. The precedence rules…
Disallow and Allow directives
Disallow prevents crawling of sections or pages; Allow permits them. Yandex lists what belongs behind Disallow: pages with confidential data,…
Robots.txtThe Sitemap directive
The Sitemap directive declares where the sitemap lives. Several may be listed, one per line: User-agent: Yandex Allow: / sitemap:…
The Crawl-delay directive
Crawl-delay has been ignored by Yandex since 22 February 2018. Any site still carrying the directive is carrying a line…
Robots.txtThe Clean-param directive
Clean-param is a Yandex extension with no equivalent elsewhere: it tells the robot which URL parameters do not change the…
Robots.txtHow to check that a robot belongs to Yandex
Robots can claim to be Yandex by setting the user agent. Verification is done by reverse DNS lookup, and the…
Robots.txtHow do I lower a load on the site or server?
The robot plans its own visits and adjusts load automatically. Requests can spike when it discovers a new section, a…
Meta tags
Meta tags tell the robot how to load, index and display a page. Yandex takes account of a specific set,…
About the IndexNow protocol
IndexNow lets a site notify Yandex directly about pages that were created, updated or deleted, instead of waiting for the…
IndexNowVerification key
The IndexNow key proves you control the site. You generate it, place a file containing it on the site, and…
IndexNow API request reference
The API is deliberately small: one endpoint, two methods. One URL — GET GET https://yandex.com/indexnow?url=<String>&key=<String>[&keyLocation=<String>] url and key are required;…
IndexNowCMS plug-ins
IndexNow can be automated with plugins written by third parties. Yandex lists them for popular systems and attaches a disclaimer…
Sitemap
Yandex supports XML and TXT sitemaps and reads them from robots.txt or from the console. Requirements worth remembering Fifty thousand…
How do I prohibit indexing of the site or pages?
Three instruments hide content from indexing, and they operate at different levels: the whole site or a section, an individual…
How do I remove pages from the search?
Removing a page from search is a two-step operation, and skipping the first makes the second fail. Prohibit indexing first,…
Site indexingSaved copy
The saved copy shows the version of a page Yandex last indexed, with the date. It is the fastest way…
Site indexingDuplicate pages
Duplicates are pages within one site serving the same content at different addresses. Yandex is precise about the boundary: pages…
Site indexingCanonical URLs
When several URLs carry the same or similar content, Yandex groups them and picks one to show — the canonical…
Site indexingProcessing redirects
Yandex recognises the standard 3xx codes: 301 and 308 as permanent, 302, 303 and 307 as temporary, and a refresh…
Site indexingSubdomain indexing
Yandex treats subdomains as separate sites for most purposes, including regional assignment — which is why regional subdomains are a…
Site indexingIndexing pages with JavaScript β
Yandex can execute JavaScript when crawling, and the console lets you decide whether it should. The setting lives under Indexing…
Site indexingIndexing localized pages
Yandex no longer reads language versions from the sitemap; localised pages are declared with alternate link markup on the page…
Site indexingIndexing AJAX sites
Yandex now crawls AJAX sites by fetching the original URLs and executing JavaScript on them. Two older mechanisms are formally…
How do I improve the site’s ranking in the search?
Yandex describes ranking as a function of several site indicators rather than a single score, and the list is a…
How do I improve the site's ranking in the search?Additional ranking signals
Yandex incorporates signals about a company’s real-world service quality — reviews, ratings, data from Yandex services — into ranking for…