{"id":23863,"date":"2007-03-06T00:00:00","date_gmt":"2007-03-06T00:00:00","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/all-about-robots\/"},"modified":"2007-03-06T00:00:00","modified_gmt":"2007-03-06T00:00:00","slug":"all-about-robots","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/all-about-robots\/","title":{"rendered":"All about robots"},"content":{"rendered":"<p class=\"gargardate\">Tuesday, March 06, 2007<\/p>\n<aside class=\"key-point\">It&#8217;s been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore). Read our <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/\">up-to-date documentation about robots.txt\/a>.<\/aside>\n<p>\n  Search engine robots, including our very own Googlebot, are incredibly polite. They work<br \/>\n  hard to respect your every wish regarding what pages they should and should not crawl. How can<br \/>\n  they tell the difference? You have to tell them, and you have to speak their language, which is<br \/>\n  an industry standard called the<br \/>\n  <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9309.html\" class=\"external-link\">Robots Exclusion Protocol<\/a>.\n<\/p>\n<p>\n  Dan Crow has written about this on the Google Blog recently, including an introduction to setting<br \/>\n  up your own rules for robots and a description of some of the more advanced options. His first<br \/>\n  two posts in the series are:\n<\/p>\n<ul>\n<li><a href=\"https:\/\/googleblog.blogspot.com\/2007\/01\/controlling-how-search-engines-access.html\" class=\"external-link\">Controlling how search engines access and index your website<\/a><\/li>\n<li><a href=\"https:\/\/googleblog.blogspot.com\/2007\/02\/robots-exclusion-protocol.html\" class=\"external-link\">The Robots Exclusion Protocol<\/a><\/li>\n<\/ul>\n<p>Stay tuned for the next installment.<\/p>\n<p>\n  Stay tuned for the next installment.\n<\/p>\n<p>\n  While we&#8217;re on the topic, I&#8217;d also like to point you to the<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/\">robots section of our help center<\/a><br \/>\n  and our earlier posts on this topic:\n<\/p>\n<ul>\n<li><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/debugging-blocked-urls\/\">Debugging Blocked URLs<\/a><\/li>\n<li><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/all-about-googlebot\/\">All About Googlebot<\/a><\/li>\n<li><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/using-a-robots-txt-file\/\">Using a robots.txt File<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Tuesday, March 06, 2007 It&#8217;s been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore). Read our up-to-date documentation about robots.txt\/a>. Search engine robots, including our very own Googlebot, are incredibly polite. They work hard [&hellip;]<\/p>\n","protected":false},"menu_order":86422,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[665],"ar_kb_etiket":[],"class_list":["post-23863","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-blog"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/23863","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/23863\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/26495"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=23863"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=23863"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=23863"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}