Robots can claim to be Yandex by setting the user agent. Verification is done by reverse DNS lookup, and the procedure is worth automating in log analysis.
The four steps
- Get the IP address from server logs.
- Reverse-lookup the IP to a host name.
- Confirm the host ends in
yandex.ru,yandex.netveyayandex.com. Any other ending means it is not Yandex. - Forward-lookup the host name and confirm it resolves back to the same IP. A mismatch means the host name is fake.
The fourth step is the one people skip, and without it the check proves nothing. The console’s IP address check tool does the same job for a single address.
Why IP ranges are not published
The robots use autonomous systems AS13238, AS208722 and AS212066 with frequently changing addresses, so no list is disclosed. Allowlisting by IP is therefore not a viable strategy; reverse DNS is the supported method.
The important detail about the robot fleet
Many Yandex robots fetch documents for purposes other than indexing, and a number of them ignore robots.txt rules written for User-agent: *. Some ignore the file entirely — YaDirectFetcher, several YandexMetrika versions. To constrain those, name them explicitly:
User-agent: YandexCalendar
Disallow: /
User-agent: YandexMobileBot
Disallow: /private/*.txt$The rule that resolves a common confusion
If one of these robots downloads a document the main indexing robot cannot access, that document is never indexed and never appears in search. Traffic in the logs from a Yandex agent is not evidence that a page is being indexed — the fleet is large and most of it is not the indexer.
Reading logs
User-agent strings include a browser version that changes over time — YandexBot/3.0 appears with a Chrome version attached. Match on the robot name, never on the full string, or log filters break silently after an update.
Biz bunu nasıl uyguluyoruz
The forward-confirmation step is the one that matters: half the “Yandex is hammering our server” reports we have looked at were other crawlers wearing the name. It is also why we match log filters on the robot name rather than the full user agent — the Chrome version inside it changes, and rules written against the whole string quietly stop matching.
İlgili hizmetler