Google’s common crawlers are used to find information for building Google’s search indexes,
perform other product specific crawls, and for analysis. They always obey
robots.txt rules
when crawling automatically. The general
technical properties
of Google’s crawlers also apply to the common crawlers.
The common crawlers generally crawl from the IP ranges published in the
common-crawlers.json object, and the reverse DNS mask
of their hostname matches crawl-***-***-***-***.googlebot.com or
geo-crawl-***-***-***-***.geo.googlebot.com.
The following list shows the common crawlers, their user agent strings as they appear in the HTTP
requests, their user agent tokens for the User-agent: line in robots.txt, and the
products that are affected by crawl preferences for the crawler. Some crawlers have more than one
user agent token; you need to match only one crawler token for a rule to apply. The list is not
exhaustive, it only covers the requestors that are more likely to show up in log files and that
we’ve received questions about.
Googlebot
User-Agentin HTTP requestsGooglebot Smartphone Googlebot Desktop Rarely:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Googlebot/2.1 (+http://www.google.com/bot.html)
robots.txt User-agent token in robots.txt GooglebotExample robots.txt group Affected products Crawling preferences addressed to the Googlebotuser agent affect Google
Search (including Discover and all Google Search features), as well as other products
such as Google Images, Google Video, Google News, and Discover.Googlebot Image
User-Agent in HTTP requests robots.txt User-agent token in robots.txt Googlebot-ImageGooglebotExample robots.txt group Affected products Crawling preferences addressed to the Googlebot-Imageuser agent affect
Google Images, Discover, Google Video, and all features in Google Search where images,
logos, and favicons are presented.Googlebot Video
User-Agent in HTTP requests robots.txt User-agent token in robots.txt Googlebot-VideoGooglebotExample robots.txt group Affected products Crawling preferences addressed to the Googlebot-Videouser agent affect
video-related Google Search features and other products dependent on videos.Googlebot News
User-Agent in HTTP requests Googlebot-news doesn’t have a separate HTTP request user agent string. Crawling is done
with various Googlebot user agent strings.robots.txt User-agent token in robots.txt Googlebot-NewsGooglebotExample robots.txt group Affected products Crawling preferences addressed to the Googlebot-Newsuser agent affect
the Google News product, including
news.google.com and the
Google News app.Google StoreBot
User-Agent in HTTP requests Desktop agent Mobile agent robots.txt User-agent token in robots.txt Storebot-GoogleExample robots.txt group Affected products Crawling preferences addressed to the Storebot-Googleuser agent affect
all surfaces of Google Shopping (for example, the Shopping tab in Google Search and
Google Shopping).Google-InspectionTool
User-Agent in HTTP requests Desktop agent Mobile agent robots.txt User-agent token in robots.txt Google-InspectionToolGooglebotExample robots.txt group Affected products Crawling preferences addressed to the Google-InspectionTooluser agent
affect Search testing tools such as the
Rich Result Test
and
URL inspection
in Search Console. It has no effect on Google Search or other products.GoogleOther
User-Agent in HTTP requests robots.txt User-agent token in robots.txt GoogleOtherExample robots.txt group Affected products Crawling preferences addressed to the GoogleOtheruser agent don’t affect
any specific product. GoogleOther is the generic crawler that may be used by various
product teams for fetching publicly accessible content from sites. For example, it may
be used for one-off crawls for internal research and development.GoogleOther-Image
User-Agent in HTTP requests robots.txt User-agent token in robots.txt GoogleOther-ImageGoogleOtherExample robots.txt group Affected products Crawling preferences addressed to the GoogleOther-Imageuser agent don’t
affect any specific product, similar to GoogleOther. GoogleOther-Image is the version
of GoogleOther optimized for fetching publicly accessible image URLs.GoogleOther-Video
User-Agent in HTTP requests robots.txt User-agent token in robots.txt GoogleOther-VideoGoogleOtherExample robots.txt group Affected products Crawling preferences addressed to the GoogleOther-Videouser agent don’t
affect any specific product, similar to GoogleOther. GoogleOther-Video is the version
of GoogleOther optimized for fetching publicly accessible video URLs.Google-CloudVertexBot
User-Agent substring in HTTP requests robots.txt User-agent token in robots.txt Google-CloudVertexBotGooglebotExample robots.txt group Affected products Crawling preferences addressed to the Google-CloudVertexBotuser agent
affect crawls requested by the site owners’ for building
Vertex AI Agents.
It has no effect on Google Search or other products.Google-Extended
User-Agent in HTTP requests Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is
done with existing Google user agent strings; the robots.txt user-agent token is used
in a control capacity.robots.txt User-agent token in robots.txt Google-ExtendedExample robots.txt group Affected products Google-Extendedis a standalone product token that web publishers can use
to manage whether content Google crawls from their sites may be used for training future
generations of Gemini models that power
Gemini Apps
and Vertex AI API for Gemini
and for grounding (providing content from the Google Search index to the model at prompt
time to improve factuality and relevancy) in
Gemini Apps
and Grounding with Google Search on Vertex AI.Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal
in Google Search.A note about Chrome/W.X.Y.Z in user agents
The string Chrome/W.X.Y.Z in the user agent strings in the list is a placeholder
that represents the version of the Chrome browser used by that user agent: for example,
41.0.2272.96. This version number increases over time to
match the latest Chromium release version used by Googlebot.If you are searching your logs or filtering your server for a user agent with this pattern, use
wildcards for the version number rather than specifying an exact version number.
How we apply this
This list is what we hand to a client’s security or CDN team. Blocking one of these by accident is a common and expensive mistake, and the list is the fastest way to settle whether a rule is safe.
Related services