{"id":23907,"date":"2007-08-15T00:00:00","date_gmt":"2007-08-15T00:00:00","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/new-robots-txt-feature-and-rep-meta-tags\/"},"modified":"2007-08-15T00:00:00","modified_gmt":"2007-08-15T00:00:00","slug":"new-robots-txt-feature-and-rep-meta-tags","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/new-robots-txt-feature-and-rep-meta-tags\/","title":{"rendered":"New robots.txt feature and REP Meta Tags"},"content":{"rendered":"

Wednesday, August 15, 2007<\/p>\n

\n We’ve improved Webmaster Central’s robots.txt analysis tool to recognize Sitemap declarations and
\n relative URLs. Earlier versions weren’t aware of Sitemaps at all, and understood only absolute
\n URLs; anything else was reported as Syntax not understood<\/code>. The improved version now
\n tells you whether your Sitemap’s URL and scope are valid. You can also test against relative
\n URLs with a lot less typing.\n<\/p>\n

\n Reporting is better, too. You’ll now be told of multiple problems per line if they exist, unlike
\n earlier versions which only reported the first problem encountered. And we’ve made other general
\n improvements to analysis and validation.\n<\/p>\n

\n Imagine that you’re responsible for the domain www.example.com<\/b> and you want search engines
\n to index everything on your site, except for your \/images<\/code> folder. You also want to
\n make sure your Sitemap gets noticed, so you save the following as your robots.txt file:\n<\/p>\n

<\/div>\n

\n You visit Webmaster Central to test your site against the robots.txt analysis tool using these two
\n test URLs:\n<\/p>\n

<\/div>\n

Earlier versions of the tool would have reported this:<\/p>\n

\"robots.txt<\/p>\n

The improved version tells you more about that robots.txt file:<\/p>\n

\"robots.txt<\/p>\n

\n See for yourself at
\n https:\/\/www.google.com\/webmasters\/tools<\/a>.\n<\/p>\n

\n We also want to make sure you’ve heard about the new unavailable_after<\/code> meta<\/code> tag
\n announced by Dan Crow on the
\n Official Google Blog<\/a>
\n a few weeks ago. This allows for a more dynamic relationship between your site and Googlebot. Just
\n think, with www.example.com<\/b>, any time you have a temporarily available news story or
\n limited offer sale or promotion page, you can specify the exact date and time you want specific
\n pages to stop being crawled and indexed.\n<\/p>\n

\n Let’s assume you’re running a promotion that expires at the end of 2007. In the headers of page
\n www.example.com\/2007promotion.html<\/b>, you would use the following:\n<\/p>\n

<\/div>\n

\n The second exciting news: the new X-Robots-Tag<\/code> rule, which adds
\n Robots Exclusion Protocol<\/a>
\n (REP) meta<\/code> tag support for non-HTML pages! Finally, you can have the same control
\n over your videos, spreadsheets, and other indexed file types. Using the example above, let’s say
\n your promotion page is in PDF format. For www.example.com\/2007promotion.pdf<\/b>, you would use
\n the following:\n<\/p>\n

<\/div>\n

\n Remember, REP meta<\/code> tags can be useful for implementing noarchive, nosnippet, and now
\n unavailable_after<\/code> tags for page-level instruction, as opposed to robots.txt, which
\n is controlled at the domain root. We get requests from bloggers and webmasters for these features,
\n so enjoy. If you have other suggestions, keep them coming. Any questions? Please ask them in the
\n Webmaster Help Group<\/a>.\n<\/p>\n

\n Posted by John Blackburn, Webmaster Tools and Matt Dougherty, Search Quality<\/p>\n","protected":false},"excerpt":{"rendered":"

Wednesday, August 15, 2007 We’ve improved Webmaster Central’s robots.txt analysis tool to recognize Sitemap declarations and relative URLs. Earlier versions weren’t aware of Sitemaps at all, and understood only absolute URLs; anything else was reported as Syntax not understood. The improved version now tells you whether your Sitemap’s URL and scope are valid. You can […]<\/p>\n","protected":false},"menu_order":86260,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[665],"ar_kb_etiket":[],"class_list":["post-23907","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-blog"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/23907","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/23907\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/26534"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=23907"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=23907"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=23907"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}