{"id":24449,"date":"2010-11-24T00:00:00","date_gmt":"2010-11-24T00:00:00","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/controlling-crawling-and-indexing-now-documented-on-code-google-com\/"},"modified":"2010-11-24T00:00:00","modified_gmt":"2010-11-24T00:00:00","slug":"controlling-crawling-and-indexing-now-documented-on-code-google-com","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/controlling-crawling-and-indexing-now-documented-on-code-google-com\/","title":{"rendered":"Controlling crawling and indexing now documented on code.google.com"},"content":{"rendered":"<p class=\"gargardate\">Wednesday, November 24, 2010<\/p>\n<p>\n  Do you know how Google&#8217;s crawler, Googlebot, handles conflicting rules in your robots.txt<br \/>\n  file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot&#8217;s favorite<br \/>\n  song? The answers to these questions (except for the last one :)), along with lots of other<br \/>\n  information about controlling the crawling and indexing of your site, are now available on<br \/>\n  <code>code.google.com<\/code>:\n<\/p>\n<p><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-syntax\/\">Controlling crawling and indexing<\/a><\/p>\n<p><img decoding=\"async\" alt=\"Googlebot doing its happy dance\" src=\"https:\/\/alienroad.com\/wp-content\/uploads\/kb-gorsel\/g-5ec79f7fb7f3.png\" loading=\"lazy\"><\/p>\n<p>\n  Now site owners have a comprehensive resource where they can learn about robots.txt files,<br \/>\n  <span>robots<\/span> <code>meta<\/code> tags, and <code>X-Robots-Tag<\/code> HTTP header rules. Please share your<br \/>\n  comments, and if you have questions you can post them in our<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/community\" class=\"external-link\">Webmaster Help Forum<\/a>.<\/p>\n<p class=\"byline-author\">\n  Posted by<br \/>\n  <a href=\"https:\/\/developers.google.com\/search\/blog\/authors\/jonathan-simon\" rel=\"author\" class=\"external-link\">Jonathan Simon<\/a>,<br \/>\n  Webmaster Trends Analyst<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Wednesday, November 24, 2010 Do you know how Google&#8217;s crawler, Googlebot, handles conflicting rules in your robots.txt file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot&#8217;s favorite song? The answers to these questions (except for the last one :)), along with lots of other information about controlling [&hellip;]<\/p>\n","protected":false},"menu_order":85063,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[665],"ar_kb_etiket":[],"class_list":["post-24449","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-blog"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/24449","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/24449\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/26826"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=24449"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=24449"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=24449"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}