{"id":23781,"date":"2006-02-06T00:00:00","date_gmt":"2006-02-06T00:00:00","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/more-stats-and-analysis-of-robots-txt-files\/"},"modified":"2006-02-06T00:00:00","modified_gmt":"2006-02-06T00:00:00","slug":"more-stats-and-analysis-of-robots-txt-files","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/more-stats-and-analysis-of-robots-txt-files\/","title":{"rendered":"More stats and analysis of robots.txt files"},"content":{"rendered":"<aside class=\"key-point\">It&#8217;s been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore).<\/aside>\n<p class=\"gargardate\">February 06, 2006<\/p>\n<p>Today, we released new features for Sitemaps.<\/p>\n<h2 id=\"robots.txt-analysis\" tabindex=\"-1\">robots.txt analysis<\/h2>\n<p>\n  If the site has a<br \/>\n  <a href=\"https:\/\/www.robotstxt.org\/wc\/exclusion.html\" class=\"external-link\">robots.txt file<\/a>,<br \/>\n  the new robots.txt tab<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/answer\/6062598\" class=\"external-link\">provides Googlebot&#8217;s view<\/a><br \/>\n  of that file, including when Googlebot last accessed it, the status it returns, and if it blocks<br \/>\n  access to your home page. This tab also lists any syntax errors in the file.\n<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/alienroad.com\/wp-content\/uploads\/kb-gorsel\/g-94fa5444d1e6.jpg\" loading=\"lazy\" alt=\"quick view of the analysed robots.txt file in webmaster tools\"><\/p>\n<p>You can enter a list of URLs to see if the robots.txt file allows or blocks them.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/alienroad.com\/wp-content\/uploads\/kb-gorsel\/g-4560134928e0.jpg\" loading=\"lazy\" alt=\"analysis of multiple urls against the robots.txt file in webmaster tools\"><\/p>\n<p>\n  You can also test changes to your robots.txt file by entering them here and then testing them<br \/>\n  against the Googlebot user-agent, other Google user agents, or the Robots Standard. This lets you<br \/>\n  experiment with changes to see how they would impact the crawl of your site, as well as make sure<br \/>\n  there are no errors in the file, before making changes to the file on your site.\n<\/p>\n<p>\n  If you don&#8217;t have a robots.txt file, you can use this page to test a potential robots.txt file<br \/>\n  before you add it to your site.\n<\/p>\n<h2 id=\"more-stats\" tabindex=\"-1\">More stats<\/h2>\n<p>\n  Crawl stats now include the page on your site that had the highest PageRank, by month, for the<br \/>\n  last three months.\n<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/alienroad.com\/wp-content\/uploads\/kb-gorsel\/g-9b451dbda707.gif\" loading=\"lazy\" alt=\"report of the blocked urls with the highest pagerank in webmaster tools\"><\/p>\n<p>\n  Page analysis now includes a list of the most common words in your site&#8217;s content and in external<br \/>\n  links to your site. This gives you additional information about why your site might come up for<br \/>\n  particular search queries.\n<\/p>\n<p class=\"byline-author\">Posted by <a href=\"https:\/\/www.vanessafox.com\/\" class=\"external-link\">Vanessa Fox<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>It&#8217;s been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore). February 06, 2006 Today, we released new features for Sitemaps. robots.txt analysis If the site has a robots.txt file, the new robots.txt tab provides [&hellip;]<\/p>\n","protected":false},"menu_order":86815,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[665],"ar_kb_etiket":[],"class_list":["post-23781","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-blog"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/23781","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/23781\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/26413"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=23781"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=23781"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=23781"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}