{"id":16566,"date":"2026-09-02T11:35:46","date_gmt":"2026-09-02T08:35:46","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-syntax\/"},"modified":"2026-09-02T16:46:02","modified_gmt":"2026-09-02T13:46:02","slug":"robots-txt-syntax","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-syntax\/","title":{"rendered":"How Google Interprets robots.txt"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Google parses the file with specific rules that differ from naive expectation. Getting them wrong usually fails silently.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Precedence<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The <strong>most specific<\/strong> rule wins, measured by path length. If <code>Disallow: \/folder\/<\/code> and <code>Allow: \/folder\/page<\/code> both match, the longer <code>Allow<\/code> wins. When rules are equally specific, the least restrictive applies.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Wildcards<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li><code>*<\/code> matches any sequence of characters.<\/li><li><code>$<\/code> anchors the end of the URL.<\/li><li><code>Disallow: \/*.pdf$<\/code> blocks PDFs but not <code>\/file.pdf?download=1<\/code>.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">User-agent groups<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A crawler obeys the single most specific group that names it, and ignores all others. A rule under <code>User-agent: *<\/code> is not merged with rules under <code>User-agent: Googlebot<\/code>, if a Googlebot group exists, the wildcard group is ignored completely for Googlebot.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Limits<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Google reads up to 500 KiB; anything past that is ignored. Comments start with <code>#<\/code>. A 4xx response is treated as &#8220;no restrictions&#8221;; a persistent 5xx is treated as &#8220;disallow everything&#8221;.<\/p>\n\n\n\n<div class=\"ar-kb-tavsiye\"><p><strong>Alien Road recommends:<\/strong> test every change in the Search Console robots.txt tester before deploying. The user-agent group rule catches people constantly, adding a Googlebot-specific line silently deactivates every wildcard rule you had, and the file still looks correct to a human reader.<\/p><\/div>\n\n\n","protected":false},"excerpt":{"rendered":"<p>Matching rules, wildcards, precedence and the size limit, the details that decide whether your file does what you think.<\/p>\n","protected":false},"menu_order":20,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[503],"ar_kb_etiket":[532,560,567],"class_list":["post-16566","ar_kb","type-ar_kb","status-publish","hentry","ar_kb_kategori-robots-txt","ar_kb_etiket-crawling","ar_kb_etiket-robots-txt","ar_kb_etiket-syntax"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16566","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16566\/revisions"}],"predecessor-version":[{"id":23698,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16566\/revisions\/23698"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=16566"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=16566"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=16566"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}