{"id":28509,"date":"2026-09-08T21:56:08","date_gmt":"2026-09-08T18:56:08","guid":{"rendered":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/the-user-agent-directive\/"},"modified":"2026-09-09T00:47:43","modified_gmt":"2026-09-08T21:47:43","slug":"the-user-agent-directive","status":"publish","type":"ar_ykb","link":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/the-user-agent-directive\/","title":{"rendered":"The User-agent directive"},"content":{"rendered":"

The robot looks for records beginning with User-agent:<\/code> and matches either the substring Yandex<\/strong> (case-insensitively) or *<\/code>. The precedence rules are strict and produce most robots.txt mistakes.<\/p>\n

The precedence rules<\/h2>\n
    \n
  • If User-agent: Yandex<\/code> is present, User-agent: *<\/code> is ignored entirely<\/strong> \u2014 not merged, ignored. Rules you assumed were global do not apply.<\/li>\n
  • If a block exists for a specific<\/em> Yandex robot, such as YandexBot<\/code>, then both the general Yandex<\/code> block and *<\/code> are unused for that robot.<\/li>\n
  • If neither Yandex<\/code> nor *<\/code> is found, the robot has unlimited access<\/strong>.<\/li>\n<\/ul>\n

    Yandex’s own example makes the trap concrete: a file with YandexBot<\/code>, Yandex<\/code> and *<\/code> blocks applies only the first to the main indexing robot, only the second to other Yandex robots, and the third to neither.<\/p>\n

    Formatting<\/h2>\n

    A blank line should precede every User-agent<\/code> directive. #<\/code> starts a comment, and everything to the end of the line is ignored \u2014 which, combined with the rule that an empty line after User-agent<\/code> detaches the rules that follow, accounts for most silently broken files.<\/p>\n

    The exception nobody expects<\/h2>\n

    Some Yandex robots may ignore robots.txt<\/code> directives altogether, including those under User-agent: Yandex<\/code>.<\/strong> To constrain those, you must name their exact user agent in its own block. This is why a site that believes it has blocked everything can still see crawling from a Yandex agent it has never heard of.<\/p>\n

    Practical shape of a file<\/h2>\n

    Write one block for Yandex<\/code> containing every rule that should apply to Yandex, and a separate complete block for *<\/code>. Do not rely on inheritance between them, because there is none. Duplication here is correct, not redundant.<\/p>\n

    Verification<\/h2>\n

    Test specific URLs per user agent in the analyser. The tool applies the same precedence rules the robot does, which is the only reliable way to confirm which block actually governs a given address.<\/p>\n","protected":false},"excerpt":{"rendered":"

    The robot looks for records beginning with User-agent: and matches either the substring Yandex (case-insensitively) or *. The precedence rules are strict and produce most robots.txt mistakes. The precedence rules If User-agent: Yandex is present, User-agent: * is ignored entirely \u2014 not merged, ignored. Rules you assumed were global do not apply. If a block […]<\/p>\n","protected":false},"menu_order":85,"template":"","meta":{"footnotes":""},"ar_ykb_kategori":[765],"ar_ykb_etiket":[],"class_list":["post-28509","ar_ykb","type-ar_ykb","status-publish","has-post-thumbnail","hentry","ar_ykb_kategori-robots-txt"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28509","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_ykb"}],"version-history":[{"count":2,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28509\/revisions"}],"predecessor-version":[{"id":29115,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28509\/revisions\/29115"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/28891"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=28509"}],"wp:term":[{"taxonomy":"ar_ykb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_kategori?post=28509"},{"taxonomy":"ar_ykb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_etiket?post=28509"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}