{"id":16565,"date":"2026-09-02T11:35:46","date_gmt":"2026-09-02T08:35:46","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/"},"modified":"2026-09-02T16:46:02","modified_gmt":"2026-09-02T13:46:02","slug":"robots-txt-intro","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/","title":{"rendered":"robots.txt: What It Controls and What It Does Not"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><code>robots.txt<\/code> tells crawlers which URLs they may request. It does <strong>not<\/strong> keep a page out of the index. A blocked URL that other pages link to can still appear in results, usually with no description, because Google was not allowed to read it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The trap<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you want a page out of Search, use <code>noindex<\/code>. But <code>noindex<\/code> only works if Google can fetch the page and read the tag. Blocking the URL in <code>robots.txt<\/code> <em>prevents<\/em> Google from seeing the <code>noindex<\/code>. The two directives cancel each other out, and the page stays indexed.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Basic rules<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>Must live at the root of the host: <code>example.com\/robots.txt<\/code>.<\/li><li>Applies per host and protocol, <code>http<\/code> and <code>https<\/code> are separate.<\/li><li>The most specific matching rule wins, not the first.<\/li><li>A missing file (404) means everything is allowed. A 5xx means Google may stop crawling entirely.<\/li><\/ul>\n\n\n\n<div class=\"ar-kb-tavsiye\"><p><strong>Alien Road recommends:<\/strong> never block CSS or JavaScript. Google renders pages before indexing them; blocked resources produce a broken render, and Google indexes what it rendered. We still find this rule in inherited files, usually copied from a template written years ago, and removing it has produced immediate coverage improvements.<\/p><\/div>\n\n\n","protected":false},"excerpt":{"rendered":"<p>The single most misunderstood file in SEO. It controls crawling, not indexing, and the difference matters.<\/p>\n","protected":false},"menu_order":10,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[503],"ar_kb_etiket":[532,530,560],"class_list":["post-16565","ar_kb","type-ar_kb","status-publish","hentry","ar_kb_kategori-robots-txt","ar_kb_etiket-crawling","ar_kb_etiket-indexing","ar_kb_etiket-robots-txt"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16565\/revisions"}],"predecessor-version":[{"id":23697,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16565\/revisions\/23697"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=16565"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=16565"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=16565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}