Crawling and indexing of locale-adaptive pages

Wednesday, January 28, 2015

Locale-adaptive pages change their content to reflect the user’s language or perceived geographic
location. Since, by default, Googlebot requests pages without setting an
Accept-Language HTTP request header and uses IP addresses that appear to be located
in the USA, not all content variants of locale-adaptive pages may be indexed completely.

Today we’re introducing new
locale-aware crawl configurations
for Googlebot for pages that we detect may adapt the content they serve based on the request’s
language and perceived location. These are:

• Geo-distributed crawling where Googlebot would start to use IP addresses that
  appear to be coming from outside the USA, in addition to the current IP addresses that appear to
  be from the USA that Googlebot currently uses.
• Language-dependent crawling where Googlebot would start to crawl with an
  Accept-Language HTTP header in the request.

As these new crawling configurations are enabled automatically for pages we detect to be
locale-adaptive, you may notice changes in how we crawl and show your site in Google search
results without you altering your CMS or server settings.

Note that these new configurations do not alter our recommendation to use separate URLs with
rel=alternate hreflang annotations
for each locale. We continue to support and recommend using separate URLs as they are still the
best way for users to interact and share your content, and also to maximize indexing and better
ranking of all variants of your content.

As always, if you have any questions or feedback, please tell us in the
internationalization Webmaster Help Forum.

Posted by Qin Yin, Software Engineer Search Infrastructure, and
Pierre Far,
Webmaster Trends Analyst