Overview of Google crawlers and fetchers (user agents)

Google uses crawlers and fetchers to perform actions for its products, either automatically or
triggered by user request. Crawler (sometimes also called a “robot” or “spider”) is a generic term
for any program that is used to
automatically discover and scan websites.
Fetchers act as a program like
wget that typically make a
single request on behalf of a user. Google’s clients fall into three categories:

Technical properties of Google’s crawlers and fetchers

Google’s crawlers and fetchers are designed to be run simultaneously by thousands of machines to
improve performance and scale as the web grows. To optimize bandwidth usage, these clients are
distributed across many datacenters across the world so they’re located near the sites that they
might access. Therefore, your logs may show visits from several IP addresses.
Google egresses primarily from IP addresses in the United States. In case Google detects that a
site is blocking requests from the United States, it may attempt to crawl from IP addresses
located in other countries.

Supported transfer protocols

Google’s crawlers and fetchers support HTTP/1.1 and
HTTP/2. The crawlers will
use the protocol version that provides the best crawling performance and may switch protocols
between crawling sessions depending on previous crawling statistics. The default protocol
version used by Google’s crawlers is HTTP/1.1; crawling over HTTP/2 may save computing resources
(for example, CPU, RAM) for your site and Googlebot, but otherwise
there’s no Google-product specific benefit to the site (for example, no ranking boost in Google Search).
To opt out from crawling over HTTP/2, instruct the server that’s hosting your site to respond
with a 421 HTTP status code when Google attempts to access your site over
HTTP/2. If that’s not feasible, you
can send a message to the Crawling team
(however this solution is temporary).

Google’s crawler infrastructure also supports crawling through FTP (as defined by
RFC959 and its
updates) and FTPS (as defined by
RFC4217 and its
updates), however crawling through these protocols is rare.

Supported content encodings

Google’s crawlers and fetchers support the following content encodings (compressions):
gzip,
deflate, and
Brotli (br). The
content encodings supported by each Google user agent is advertised in the
Accept-Encoding header of each request they make. For example,
Accept-Encoding: gzip, deflate, br.

File size limits

By default, Google’s crawlers and fetchers only crawl the first 15MB of a file, and any content
beyond this limit is ignored. However, individual projects may set different limits for their crawlers and
fetchers, and also for different file types. For example, a Google crawler
like Googlebot may
have a smaller size limit (for example, 2MB), or specify a larger file size limit for a PDF than for HTML.

Crawl rate and host load

Our goal
is to crawl as many pages from your site as we can on each visit without overwhelming your
server. If your site is having trouble keeping up with Google’s crawling requests, you can
reduce the crawl rate. Note that
sending the inappropriate
HTTP response code
to Google’s crawlers may affect how your site appears in Google products.

HTTP Caching

Google’s crawling infrastructure supports heuristic HTTP caching as defined by the
HTTP caching standard,
specifically through the ETag response- and If-None-Match request
header, and the Last-Modified response- and If-Modified-Since request
header.

If both ETag and Last-Modified response header fields are present in the
HTTP response, Google’s crawlers use the ETag value as
required by the HTTP standard.
For Google’s crawlers specifically, we recommend using
ETag
instead of the Last-Modified header to indicate caching preference as
ETag doesn’t have date formatting issues.

Other HTTP caching directives aren’t supported.

Individual Google crawlers and fetchers may or may not make use of caching, depending on the needs
of the product they’re associated with. For example, Googlebot supports caching when
re-crawling URLs for Google Search, and Storebot-Google only supports caching in
certain conditions.

To implement HTTP caching for your site, get in touch with your hosting or content management
system provider.

ETag and If-None-Match

Google’s crawling infrastructure supports ETag and If-None-Match as
defined by the
HTTP Caching standard.
Learn more about the
ETag
response header and its request header counterpart,
If-None-Match.

Last-Modified and If-Modified-Since

Google’s crawling infrastructure supports Last-Modified and
If-Modified-Since as defined by the
HTTP Caching standard
with the following caveats:

• The date in the Last-Modified header must be formatted according to the
  HTTP standard.
  To avoid parsing issues, we recommend using the following date format:
  “Weekday, DD Mon YYYY HH:MM:SS Timezone”. For example,
  “Fri, 4 Sep 1998 19:15:56 GMT“.
• While not required, consider also setting the
  max-age field of the Cache-Control response header
  to help crawlers determine when to recrawl the specific URL. Set the value of the
  max-age field to the expected number of seconds the content will be unchanged. For
  example, Cache-Control: max-age=94043.

Learn more about the
Last-Modified
response header and its request header counterpart, If-Modified-Since.

Verify Google’s crawlers and fetchers

Google’s crawlers identify themselves in three ways:

1. The HTTP user-agent request header.
2. The source IP address of the request.
3. The reverse DNS hostname of the source IP.

Learn how to use these details to
verify Google’s crawlers and fetchers.