{"id":28508,"date":"2026-09-08T21:56:08","date_gmt":"2026-09-08T18:56:08","guid":{"rendered":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/using-robots-txt\/"},"modified":"2026-09-09T00:47:43","modified_gmt":"2026-09-08T21:47:43","slug":"using-robots-txt","status":"publish","type":"ar_ykb","link":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/using-robots-txt\/","title":{"rendered":"Using robots.txt"},"content":{"rendered":"

robots.txt<\/code> sets crawling parameters for search robots. Restricting sections reduces server load and speeds the site up. Yandex supports the Robots Exclusion Protocol with its own extensions.<\/p>\n

The warning that comes first, and matters most<\/h2>\n

Pages restricted in robots.txt<\/code> can still appear in Yandex search.<\/strong> To remove a page from results you must use the noindex<\/code> directive in the HTML or the HTTP header \u2014 and you must not block that page in robots.txt<\/code><\/em>, because a blocked page is never fetched and the noindex<\/code> is never seen.<\/p>\n

This is the most consequential misunderstanding in technical SEO, and Yandex states it in the first paragraph. Blocking is about crawl budget; noindex<\/code> is about search presence. Using the first to achieve the second guarantees the page stays indexed permanently.<\/p>\n

File requirements<\/h2>\n
    \n
  • Maximum 500 KB<\/strong>.<\/li>\n
  • A TXT file named robots.txt<\/code>.<\/li>\n
  • In the root directory<\/strong>.<\/li>\n
  • Accessible to robots, returning 200 OK<\/code>.<\/li>\n<\/ul>\n

    If the file does not meet these requirements, the site is considered open for indexing.<\/strong> A malformed or unreachable file is not a safety net \u2014 it is an open door.<\/p>\n

    The redirect feature<\/h2>\n

    Yandex accepts a redirect from one site’s robots.txt<\/code> to another site’s, and applies the directives in the target file. That is specifically useful during a site move<\/strong>, where both domains must present consistent rules \u2014 and it is why the migration guidance insists the two files agree.<\/p>\n

    The supported directives<\/h2>\n
      \n
    • User-agent<\/code><\/strong> \u2014 which robot the rules apply to.<\/li>\n
    • Disallow<\/code><\/strong> \u2014 prohibit crawling of sections or pages.<\/li>\n
    • Sitemap<\/code><\/strong> \u2014 the path to the sitemap, given in full with protocol.<\/li>\n
    • Clean-param<\/code><\/strong> \u2014 a Yandex extension declaring parameters to ignore.<\/li>\n<\/ul>\n

      Verification<\/h2>\n

      Test the file in the console’s robots.txt analyser with actual URLs, not just for syntax. A file that parses cleanly and blocks the wrong directory reports no errors at all.<\/p>\n","protected":false},"excerpt":{"rendered":"

      robots.txt sets crawling parameters for search robots. Restricting sections reduces server load and speeds the site up. Yandex supports the Robots Exclusion Protocol with its own extensions. The warning that comes first, and matters most Pages restricted in robots.txt can still appear in Yandex search. To remove a page from results you must use the […]<\/p>\n","protected":false},"menu_order":84,"template":"","meta":{"footnotes":""},"ar_ykb_kategori":[765],"ar_ykb_etiket":[],"class_list":["post-28508","ar_ykb","type-ar_ykb","status-publish","has-post-thumbnail","hentry","ar_ykb_kategori-robots-txt"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28508","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_ykb"}],"version-history":[{"count":2,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28508\/revisions"}],"predecessor-version":[{"id":29114,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28508\/revisions\/29114"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/28890"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=28508"}],"wp:term":[{"taxonomy":"ar_ykb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_kategori?post=28508"},{"taxonomy":"ar_ykb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_etiket?post=28508"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}