{"id":28510,"date":"2026-09-08T21:56:08","date_gmt":"2026-09-08T18:56:08","guid":{"rendered":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/disallow-and-allow-directives\/"},"modified":"2026-09-09T00:47:43","modified_gmt":"2026-09-08T21:47:43","slug":"disallow-and-allow-directives","status":"publish","type":"ar_ykb","link":"https:\/\/alienroad.com\/yandex-bilgi-bankasi\/disallow-and-allow-directives\/","title":{"rendered":"Disallow and Allow directives"},"content":{"rendered":"<p><code>Disallow<\/code> prevents crawling of sections or pages; <code>Allow<\/code> permits them. Yandex lists what belongs behind <code>Disallow<\/code>: pages with confidential data, internal search results, traffic statistics, duplicates, logs, and database service pages.<\/p>\n<h2>The recommendation people miss<\/h2>\n<p>For pages whose addresses carry <strong>GET parameters<\/strong>, Yandex recommends <strong><code>Clean-param<\/code> rather than <code>Disallow<\/code><\/strong>, and explains why: with <code>Disallow<\/code>, it may be unable to identify the duplicate&#8217;s parameter-free counterpart, and <em>some of the blocked pages&#8217; accumulated metrics are lost<\/em>. Blocking discards signals; <code>Clean-param<\/code> consolidates them onto the canonical address.<\/p>\n<p>This is a genuinely different answer from the one most technical audits give, and it applies to the largest category of unwanted URLs on any commercial site \u2014 tracking tags, sort orders, session identifiers.<\/p>\n<h2>Patterns<\/h2>\n<pre><code>Disallow: \/            # the whole site\nDisallow: \/catalogue   # anything beginning \/catalogue\nDisallow: \/page?       # anything whose URL contains parameters after \/page<\/code><\/pre>\n<h2>Allow in combination<\/h2>\n<pre><code>User-agent: Yandex\nAllow: \/cgi-bin\nDisallow: \/<\/code><\/pre>\n<p>Everything blocked except paths beginning <code>\/cgi-bin<\/code>. This inversion \u2014 block all, permit specific \u2014 is the right shape for a staging environment or an application where only a few sections should ever be crawled, and it is far safer than enumerating every directory to exclude.<\/p>\n<h2>What Disallow does not do<\/h2>\n<p>It does not remove a page from search. A blocked page can still be listed. Worse, blocking a page prevents the robot from reading a <code>noindex<\/code> on it, so the two instruments must never be combined on the same URL.<\/p>\n<h2>Before deploying<\/h2>\n<p>Run the intended URLs through the analyser \u2014 both those that should be blocked and those that must not be. A rule of the wrong precision is invisible until traffic disappears.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Disallow prevents crawling of sections or pages; Allow permits them. Yandex lists what belongs behind Disallow: pages with confidential data, internal search results, traffic statistics, duplicates, logs, and database service pages. The recommendation people miss For pages whose addresses carry GET parameters, Yandex recommends Clean-param rather than Disallow, and explains why: with Disallow, it may [&hellip;]<\/p>\n","protected":false},"menu_order":86,"template":"","meta":{"footnotes":""},"ar_ykb_kategori":[765],"ar_ykb_etiket":[],"class_list":["post-28510","ar_ykb","type-ar_ykb","status-publish","has-post-thumbnail","hentry","ar_ykb_kategori-robots-txt"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28510","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_ykb"}],"version-history":[{"count":2,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28510\/revisions"}],"predecessor-version":[{"id":29116,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb\/28510\/revisions\/29116"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/28892"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=28510"}],"wp:term":[{"taxonomy":"ar_ykb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_kategori?post=28510"},{"taxonomy":"ar_ykb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_ykb_etiket?post=28510"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}