{"id":24843,"date":"2014-07-16T00:00:00","date_gmt":"2014-07-16T00:00:00","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/testing-robots-txt-files-made-easier\/"},"modified":"2014-07-16T00:00:00","modified_gmt":"2014-07-16T00:00:00","slug":"testing-robots-txt-files-made-easier","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/testing-robots-txt-files-made-easier\/","title":{"rendered":"Testing robots.txt files made easier"},"content":{"rendered":"<p class=\"gargardate\">Wednesday, July 16, 2014<\/p>\n<aside class=\"key-point\">It&#8217;s been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore). Check out the newer <a href=\"https:\/\/support.google.com\/webmasters\/answer\/6062598\" class=\"external-link\">robots.txt report<\/a>.<\/aside>\n<p>To crawl, or not to crawl, that is the robots.txt question.<\/p>\n<p>\n  Making and maintaining correct robots.txt files can sometimes be difficult. While most sites have<br \/>\n  it easy (tip: they often don&#8217;t even need a robots.txt file!), finding the rules within a<br \/>\n  large robots.txt file that are or were blocking individual URLs can be quite tricky. To make that<br \/>\n  easier, we&#8217;re now announcing an updated<br \/>\n  <a href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\" class=\"external-link\">robots.txt testing tool<\/a><br \/>\n  in Webmaster Tools.\n<\/p>\n<p>\n  You can find the updated testing tool in<br \/>\n  <a href=\"https:\/\/search.google.com\/search-console\" class=\"external-link\">Webmaster Tools<\/a><br \/>\n  within the Crawl section:\n<\/p>\n<p><img decoding=\"async\" alt src=\"https:\/\/alienroad.com\/wp-content\/uploads\/kb-gorsel\/g-b0475776cf9f.png\" loading=\"lazy\"><\/p>\n<p>\n  Here you&#8217;ll see the current robots.txt file, and can test new URLs to see whether they&#8217;re<br \/>\n  disallowed for crawling. To guide your way through complicated rules, it will highlight the<br \/>\n  specific one that led to the final decision. You can make changes in the file and test those too,<br \/>\n  you&#8217;ll just need to upload the new version of the file to your server afterwards to make the<br \/>\n  changes take effect. Our developers site has<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/\">more about robots.txt rules and how the files are processed<\/a>.\n<\/p>\n<p>\n  Additionally, you&#8217;ll be able to review older versions of your robots.txt file, and see when access<br \/>\n  issues block us from crawling. For example, if Googlebot sees a <code>500<\/code> server error for<br \/>\n  the robots.txt file, we&#8217;ll generally pause further crawling of the website.\n<\/p>\n<p>\n  Since there may be some errors or warnings shown for your existing sites, we recommend<br \/>\n  double-checking their robots.txt files. You can also combine it with other parts of Webmaster<br \/>\n  Tools: for example, you might use the updated<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/answer\/158587\" class=\"external-link\">Fetch as Google tool<\/a><br \/>\n  to render important pages on your website. If any blocked URLs are reported, you can use this<br \/>\n  robots.txt tester to find the rule that&#8217;s blocking them, and, of course, then improve that.<br \/>\n  A common problem we&#8217;ve seen comes from old robots.txt files that block CSS, JavaScript, or mobile<br \/>\n  content&mdash;fixing that is often trivial once you&#8217;ve seen it.\n<\/p>\n<p>\n  We hope this updated tool makes it easier for you to test and maintain the robots.txt file. Should<br \/>\n  you have any questions, or need help with crafting a good set of rules, you can drop by our<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/go\/community\" class=\"external-link\">webmaster&#8217;s help forum<\/a>!\n<\/p>\n<p class=\"byline-author\">Posted by <a href=\"https:\/\/www.linkedin.com\/in\/asaph-arnon-4627987\/\" class=\"external-link\">Asaph Arnon<\/a>, Webmaster Tools team<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Wednesday, July 16, 2014 It&#8217;s been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore). Check out the newer robots.txt report. To crawl, or not to crawl, that is the robots.txt question. Making and maintaining [&hellip;]<\/p>\n","protected":false},"menu_order":83733,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[665],"ar_kb_etiket":[],"class_list":["post-24843","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-blog"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/24843","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/24843\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27032"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=24843"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=24843"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=24843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}