Tuesday, October 06, 2009<\/p>\n
\n Handling duplicate content within your own website can be a big challenge. Websites grow; features
\n get added, changed and removed; content comes\u2014content goes. Over time, many websites collect
\n systematic cruft in the form of multiple URLs that return the same contents. Having duplicate
\n content on your website is generally not problematic, though it can make it harder for search
\n engines to crawl and index the content. Also, PageRank and similar information found via incoming
\n links can get diffused across pages we aren’t currently recognizing as duplicates, potentially
\n making your preferred version of the page rank lower in Google.\n<\/p>\n
site:<\/code> query<\/a>
\n in Google. Multiple results for the same content show duplication you can investigate.\n <\/li>\n- \n Determine your preferred URLs.<\/b> Before fixing duplicate content issues, you’ll have to
\n determine your preferred URL structure. Which URL would you prefer to use for that piece of
\n content?\n <\/li>\n- \n Be consistent within your website.<\/b> Once you’ve chosen your preferred URLs, make sure to
\n use them in all possible locations within your website (including in your
\n Sitemap file<\/a>).\n <\/li>\n- \n Apply
301<\/code> permanent redirects where necessary and possible.<\/b> If you can,
\n redirect duplicate URLs to your preferred URLs using a 301<\/code> response code. This
\n helps users and search engines find your preferred URLs should they visit the duplicate URLs.
\n If your site is available on several domain names, pick one and use the 301<\/code>
\n redirect appropriately from the others, making sure to forward to the right specific page, not
\n just the root of the domain. If you support both www and non-www host names, pick one, use the
\n preferred domain setting in Webmaster Tools<\/a>,
\n and redirect appropriately.\n <\/li>\n- \n Implement
\n the rel=\"canonical\"<\/code> link element<\/a>
\n on your pages where you can.
\n <\/b> Where 301<\/code> redirects are not possible, the rel=\"canonical\"<\/code> link
\n element can give us a better understanding of your site and of your preferred URLs. The use of
\n this link element is also supported by major search engines such as
\n Ask.com<\/a>,
\n Bing<\/a>
\n and
\n Yahoo!<\/a>.\n <\/li>\n- \n Use the
\n URL parameter handling tool<\/a>
\n in Google Webmaster Tools where possible.
\n <\/b> If some or all of your website’s duplicate content comes from URLs with query parameters,
\n this tool can help you to notify us of important and irrelevant parameters within your URLs.
\n More information about this tool can be found in our
\n announcement blog post<\/a>.\n <\/li>\n<\/ol>\nWhat about the robots.txt file?<\/h2>\n
\n One item which is missing from this list is disallowing crawling of duplicate content with your
\n robots.txt file. We now recommend not blocking access to duplicate content on your website,
\n whether with a robots.txt file or other methods<\/b>. Instead, use the
\n rel=\"canonical\"<\/code> link element<\/a>,
\n the
\n URL parameter handling tool<\/a>,
\n o 301<\/code> redirects. If access to duplicate content is entirely blocked, search engines
\n effectively have to treat those URLs as separate, unique pages since they cannot know that they’re
\n actually just different URLs for the same content. A better solution is to allow them to be
\n crawled, but clearly mark them as duplicate using one of our recommended methods. If you allow us
\n to crawl these URLs, Googlebot will learn rules to identify duplicates just by looking at the URL
\n and should largely avoid unnecessary recrawls in any case. In cases where duplicate content still
\n leads to us crawling too much of your website, you can also
\n adjust the crawl rate setting in Webmaster Tools<\/a>.\n<\/p>\n\n We hope these methods will help you to manage the duplicate content on your website! Information
\n about duplicate content in general can also be found in our
\n Help Center<\/a>. Should you have any
\n questions, you can join the discussion in our
\n Webmaster Help Forum<\/a>.\n<\/p>\n