If your site has locale-adaptive pages (that is, your site returns different content
based on the perceived country or preferred language of the visitor), Google might not crawl,
index, or rank all your content for different locales. This is because the default IP
addresses of the Googlebot crawler appear to be based in the USA. In addition, the crawler
sends HTTP requests without setting Accept-Language in the request header.
Geo-distributed crawling
Googlebot crawls with IP addresses based outside the USA, in addition to the US-based IP addresses.
As we have always recommended, when Googlebot appears to come from a certain country, treat
it like you would treat any other user from that country. This means that if you block
USA-based users from accessing your content, but allow visitors from Australia to see it,
your server should block Googlebot if it appears to be coming from the USA, but allow access
to Googlebot if it appears to come from Australia.
Other considerations
- Googlebot uses the same user agent string for all crawling configurations. Learn more about
the user agent strings used
by Google crawlers. - You can verify Googlebot
geo-distributed crawls using reverse DNS lookups. - If your site is using the robots exclusion protocol,
make sure you apply it consistently
across locales. This means that robotsmetatags
and the robots.txt file must
specify the same rules in each locale.
Comment nous l'appliquons
Locale-adaptive serving is fragile because Googlebot crawls from a limited set of locations. Where a client insists on it, we make sure the fallback experience is the one Google sees, and we prefer separate URLs whenever the business can accept them.
Services associés