{"id":25661,"date":"2026-09-06T00:46:30","date_gmt":"2026-09-05T21:46:30","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/overview-of-google-crawlers-and-fetchers-user-agents\/"},"modified":"2026-09-06T00:49:54","modified_gmt":"2026-09-05T21:49:54","slug":"overview-of-google-crawlers-and-fetchers-user-agents","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/overview-of-google-crawlers-and-fetchers-user-agents\/","title":{"rendered":"Overview of Google crawlers and fetchers (user agents)"},"content":{"rendered":"<p>\n  Google uses crawlers and fetchers to perform actions for its products, either automatically or<br \/>\n  triggered by user request. Crawler (sometimes also called a &#8220;robot&#8221; or &#8220;spider&#8221;) is a generic term<br \/>\n  for any program that is used to<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-search-works\/#crawling\" class=\"external-link\">automatically discover and scan websites<\/a>.<br \/>\n  Fetchers act as a program like<br \/>\n  <a href=\"https:\/\/www.gnu.org\/software\/wget\/\" class=\"external-link\">wget<\/a> that typically make a<br \/>\n  single request on behalf of a user. Google&#8217;s clients fall into three categories:\n<\/p>\n<aside class=\"tip\">\n  <b>Looking for the latest updates to this page?<\/b><br \/>\n  See our <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/changelog\/\">updates to our documentation<\/a>.<br \/>\n<\/aside>\n<table id=\"crawler-types\">\n<tr>\n<td><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-googles-common-crawlers\/\">Common crawlers<\/a><\/td>\n<td>\n      The common crawlers used for Google&#8217;s products (such as<br \/>\n      <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/googlebot\/\" class=\"external-link\">Googlebot<\/a>). They<br \/>\n      always respect robots.txt rules for automatic crawls.\n    <\/td>\n<\/tr>\n<tr>\n<td><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-googles-special-case-crawlers\/\">Special-case crawlers<\/a><\/td>\n<td>\n      Special-case crawlers are similar to common crawlers, however are used by specific products<br \/>\n      where there&#8217;s an agreement between the crawled site and the Google product about the crawl<br \/>\n      process. For example, <code>AdsBot<\/code> ignores the global robots.txt user agent<br \/>\n      (<code>*<\/code>) with the ad publisher&#8217;s permission.\n    <\/td>\n<\/tr>\n<tr>\n<td><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-google-user-triggered-fetchers\/\">User-triggered fetchers<\/a><\/td>\n<td>\n      User-triggered fetchers are part of tools and product functions where the end user triggers a<br \/>\n      fetch. For example,<br \/>\n      <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9008080\" class=\"external-link\">Google Site Verifier<\/a><br \/>\n      acts on the request of a user.\n    <\/td>\n<\/tr>\n<\/table>\n<h2 id=\"crawl-technical-props\" tabindex=\"-1\">Technical properties of Google&#8217;s crawlers and fetchers<\/h2>\n<p>\n  Google&#8217;s crawlers and fetchers are designed to be run simultaneously by thousands of machines to<br \/>\n  improve performance and scale as the web grows. To optimize bandwidth usage, these clients are<br \/>\n  distributed across many datacenters across the world so they&#8217;re located near the sites that they<br \/>\n  might access. Therefore, your logs may show visits from several IP addresses.<br \/>\n  Google egresses primarily from IP addresses in the United States. In case Google detects that a<br \/>\n  site is blocking requests from the United States, it may attempt to crawl from IP addresses<br \/>\n  located in other countries.\n<\/p>\n<h3 id=\"supported-transfer-protocols\" tabindex=\"-1\">Supported transfer protocols<\/h3>\n<p>\n  Google&#8217;s crawlers and fetchers support HTTP\/1.1 and<br \/>\n  <a href=\"https:\/\/en.wikipedia.org\/wiki\/HTTP\/2\" class=\"external-link\">HTTP\/2<\/a>. The crawlers will<br \/>\n  use the protocol version that provides the best crawling performance and may switch protocols<br \/>\n  between crawling sessions depending on previous crawling statistics. The default protocol<br \/>\n  version used by Google&#8217;s crawlers is HTTP\/1.1; crawling over HTTP\/2 may save computing resources<br \/>\n  (for example, CPU, RAM) for your site and Googlebot, but otherwise<br \/>\n  there&#8217;s no Google-product specific benefit to the site (for example, no ranking boost in Google Search).<br \/>\n  To opt out from crawling over HTTP\/2, instruct the server that&#8217;s hosting your site to respond<br \/>\n  with a <code>421<\/code> HTTP status code when Google attempts to access your site over<br \/>\n  HTTP\/2. If that&#8217;s not feasible, you<br \/>\n  <a href=\"https:\/\/www.google.com\/webmasters\/tools\/googlebot-report\" class=\"external-link\">can send a message to the Crawling team<\/a><br \/>\n  (however this solution is temporary).\n<\/p>\n<p>\n  Google&#8217;s crawler infrastructure also supports crawling through FTP (as defined by<br \/>\n  <a href=\"https:\/\/datatracker.ietf.org\/doc\/html\/rfc959\" class=\"external-link\">RFC959<\/a> and its<br \/>\n  updates) and FTPS (as defined by<br \/>\n  <a href=\"https:\/\/datatracker.ietf.org\/doc\/html\/rfc4217\" class=\"external-link\">RFC4217<\/a> and its<br \/>\n  updates), however crawling through these protocols is rare.\n<\/p>\n<h3 id=\"supported-content-encodings\" tabindex=\"-1\">Supported content encodings<\/h3>\n<p>\n  Google&#8217;s crawlers and fetchers support the following content encodings (compressions):<br \/>\n  <a href=\"https:\/\/en.wikipedia.org\/wiki\/Gzip\" class=\"external-link\">gzip<\/a>,<br \/>\n  <a href=\"https:\/\/en.wikipedia.org\/wiki\/Deflate\" class=\"external-link\">deflate<\/a>, and<br \/>\n  <a href=\"https:\/\/en.wikipedia.org\/wiki\/Brotli\" class=\"external-link\">Brotli (br)<\/a>. The<br \/>\n  content encodings supported by each Google user agent is advertised in the<br \/>\n  <code>Accept-Encoding<\/code> header of each request they make. For example,<br \/>\n  <code>Accept-Encoding: gzip, deflate, br<\/code>.\n<\/p>\n<h3 id=\"file-size-limits\" tabindex=\"-1\">File size limits<\/h3>\n<p>\n  By default, Google&#8217;s crawlers and fetchers only crawl the first 15MB of a file, and any content<br \/>\n  beyond this limit is ignored. However, individual projects may set different limits for their crawlers and<br \/>\n  fetchers, and also for different file types. For example, a Google crawler<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/googlebot\/\" class=\"external-link\">like Googlebot<\/a> may<br \/>\n  have a smaller size limit (for example, 2MB), or specify a larger file size limit for a PDF than for HTML.\n<\/p>\n<h3 id=\"crawl-rate-and-host-load\" tabindex=\"-1\">Crawl rate and host load<\/h3>\n<p>\n  Our goal<br \/>\n  is to crawl as many pages from your site as we can on each visit without overwhelming your<br \/>\n  server. If your site is having trouble keeping up with Google&#8217;s crawling requests, you can<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/reduce-the-google-crawl-rate\/\">reduce the crawl rate<\/a>. Note that<br \/>\n  sending the inappropriate<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-http-status-codes-affect-googles-crawlers\/\" class=\"external-link\">HTTP response code<\/a><br \/>\n  to Google&#8217;s crawlers may affect how your site appears in Google products.\n<\/p>\n<h3 id=\"http-caching\" tabindex=\"-1\">HTTP Caching<\/h3>\n<p>\n  Google&#8217;s crawling infrastructure supports heuristic HTTP caching as defined by the<br \/>\n  <a href=\"https:\/\/httpwg.org\/specs\/rfc9111.html\" class=\"external-link\">HTTP caching standard<\/a>,<br \/>\n  specifically through the <code>ETag<\/code> response- and <code>If-None-Match<\/code> request<br \/>\n  header, and the <code>Last-Modified<\/code> response- and <code>If-Modified-Since<\/code> request<br \/>\n  header.\n<\/p>\n<aside class=\"note\">\n  Note: Consider setting both the <code>Etag<\/code> and <code>Last-Modified<\/code> values regardless<br \/>\n  of the preference of Google&#8217;s crawlers. These headers are also used by other applications such<br \/>\n  as CMSes.<br \/>\n<\/aside>\n<p>\n  If both <code>ETag<\/code> and <code>Last-Modified<\/code> response header fields are present in the<br \/>\n  HTTP response, Google&#8217;s crawlers use the <code>ETag<\/code> value as<br \/>\n  <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110.html#section-13.1.3\" class=\"external-link\">required by the HTTP standard<\/a>.<br \/>\n  For Google&#8217;s crawlers specifically, we recommend using<br \/>\n  <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110#name-etag\" class=\"external-link\"><code>ETag<\/code><\/a><br \/>\n  instead of the <code>Last-Modified<\/code> header to indicate caching preference as<br \/>\n  <code>ETag<\/code> doesn&#8217;t have date formatting issues.\n<\/p>\n<p>\n  Other HTTP caching directives aren&#8217;t supported.\n<\/p>\n<p>\n  Individual Google crawlers and fetchers may or may not make use of caching, depending on the needs<br \/>\n  of the product they&#8217;re associated with. For example, <code>Googlebot<\/code> supports caching when<br \/>\n  re-crawling URLs for Google Search, and <code>Storebot-Google<\/code> only supports caching in<br \/>\n  certain conditions.\n<\/p>\n<p>\n  To implement HTTP caching for your site, get in touch with your hosting or content management<br \/>\n  system provider.\n<\/p>\n<h4 id=\"etag-and-if-none-match\" tabindex=\"-1\"><code>ETag<\/code> and <code>If-None-Match<\/code><\/h4>\n<p>\n  Google&#8217;s crawling infrastructure supports <code>ETag<\/code> and <code>If-None-Match<\/code> as<br \/>\n  defined by the<br \/>\n  <a href=\"https:\/\/httpwg.org\/specs\/rfc9111.html\" class=\"external-link\">HTTP Caching standard<\/a>.<br \/>\n  Learn more about the<br \/>\n  <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110#name-etag\" class=\"external-link\"><code>ETag<\/code><\/a><br \/>\n  response header and its request header counterpart,<br \/>\n  <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110#name-if-none-match\" class=\"external-link\"><code>If-None-Match<\/code><\/a>.\n<\/p>\n<h4 id=\"last-modified-and-if-modified-since\" tabindex=\"-1\">Last-Modified and If-Modified-Since<\/h4>\n<p>\n  Google&#8217;s crawling infrastructure supports <code>Last-Modified<\/code> and<br \/>\n  <code>If-Modified-Since<\/code> as defined by the<br \/>\n  <a href=\"https:\/\/httpwg.org\/specs\/rfc9111.html\" class=\"external-link\">HTTP Caching standard<\/a><br \/>\n  with the following caveats:\n<\/p>\n<ul>\n<li>\n    The date in the <code>Last-Modified<\/code> header must be formatted according to the<br \/>\n    <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110.html\" class=\"external-link\">HTTP standard<\/a>.<br \/>\n    To avoid parsing issues, we recommend using the following date format:<br \/>\n    &#8220;Weekday, <span>DD Mon YYYY HH:MM:SS<\/span> Timezone&#8221;. For example,<br \/>\n    &#8220;<span>Fri, 4 Sep 1998 19:15:56 GMT<\/span>&#8220;.\n  <\/li>\n<li>\n    While not required, consider also setting the<br \/>\n    <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9111.html#name-max-age-2\" class=\"external-link\"><code>max-age<\/code> field of the <code>Cache-Control<\/code> response header<\/a><br \/>\n    to help crawlers determine when to recrawl the specific URL. Set the value of the<br \/>\n    <code>max-age<\/code> field to the expected number of seconds the content will be unchanged. For<br \/>\n    example, <code>Cache-Control: max-age=94043<\/code>.\n  <\/li>\n<\/ul>\n<p>\n  Learn more about the<br \/>\n  <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110#name-last-modified\" class=\"external-link\"><code>Last-Modified<\/code><\/a><br \/>\n  response header and its request header counterpart, <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110#name-if-modified-since\" class=\"external-link\"><code>If-Modified-Since<\/code><\/a>.\n<\/p>\n<h2 id=\"crawl-verify-crawler\" tabindex=\"-1\">Verify Google&#8217;s crawlers and fetchers<\/h2>\n<p>\n  Google&#8217;s crawlers identify themselves in three ways:\n<\/p>\n<ol>\n<li>\n    The HTTP <code>user-agent<\/code> request header.\n  <\/li>\n<li>\n    The source IP address of the request.\n  <\/li>\n<li>\n    The reverse DNS hostname of the source IP.\n  <\/li>\n<\/ol>\n<p>\n  Learn how to use these details to<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/verify-requests-from-google-crawlers-and-fetchers\/\">verify Google&#8217;s crawlers and fetchers<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Understand the technical properties of Google crawlers and fetchers, including supported transfer protocols, caching, and file size limits.<\/p>\n","protected":false},"menu_order":7000,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[708],"ar_kb_etiket":[],"class_list":["post-25661","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-crawling-infrastructure-crawling-and-indexing"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25661","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25661\/revisions"}],"predecessor-version":[{"id":25706,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25661\/revisions\/25706"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27658"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25661"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25661"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25661"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}