Wednesday, November 24, 2010
Do you know how Google’s crawler, Googlebot, handles conflicting rules in your robots.txt
file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot’s favorite
song? The answers to these questions (except for the last one :)), along with lots of other
information about controlling the crawling and indexing of your site, are now available on
code.google.com:
Controlling crawling and indexing
Now site owners have a comprehensive resource where they can learn about robots.txt files,
robots meta tags, and X-Robots-Tag HTTP header rules. Please share your
comments, and if you have questions you can post them in our
Webmaster Help Forum.
How we apply this
We archive Search Central announcements because client questions often trace back to a change that was announced years ago. Reading the original is faster than reconstructing it from second-hand summaries.
Related services