{"id":25668,"date":"2026-09-06T00:47:43","date_gmt":"2026-09-05T21:47:43","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-googles-special-case-crawlers\/"},"modified":"2026-09-06T00:51:36","modified_gmt":"2026-09-05T21:51:36","slug":"list-of-googles-special-case-crawlers","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-googles-special-case-crawlers\/","title":{"rendered":"List of Google&#8217;s special-case crawlers"},"content":{"rendered":"<p>\n  The special-case crawlers are used by specific Google products where there&#8217;s an agreement between<br \/>\n  the crawled site and the product about the crawl process. For example, <code>AdsBot<\/code> ignores<br \/>\n  the global robots.txt user agent (<code>*<\/code>) with the ad publisher&#8217;s permission. The general<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/overview-of-google-crawlers-and-fetchers-user-agents\/#crawl-technical-props\">technical properties<\/a><br \/>\n  of Google&#8217;s crawlers also apply to the special-case crawlers.\n<\/p>\n<p>\n  The<br \/>\n  special-case crawlers may ignore robots.txt rules and so they operate from a different IP range<br \/>\n  than the common crawlers. The IP ranges are published in the<br \/>\n  <a href=\"https:\/\/developers.google.com\/static\/crawling\/ipranges\/special-crawlers.json\">special-crawlers.json<\/a> object. The<br \/>\n  special-case crawlers&#8217; reverse DNS mask matches<br \/>\n  <code>rate-limited-proxy-***-***-***-***.google.com<\/code>.\n<\/p>\n<p>\n  The following list shows the special-case crawlers, their user agent strings as they appear in<br \/>\n  the HTTP requests, their user agent tokens for the <code>User-agent:<\/code> line in robots.txt,<br \/>\n  and the products that are affected by crawl preferences for the crawler. The list is not<br \/>\n  exhaustive, it only covers the requestors that are more likely to show up in log files and that<br \/>\n  we&#8217;ve received questions about.\n<\/p>\n<aside class=\"caution\">\n  <b>Caution<\/b>: The user agent string can be spoofed.<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/verify-requests-from-google-crawlers-and-fetchers\/\">Learn how to verify if a requestor is from Google<\/a>.<br \/>\n<\/aside>\n<dl>\n<dt>\n<h2 class=\"add-link\" id=\"apis-google\" tabindex=\"-1\"><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/apis-google-user-agent\/\">APIs-Google<\/a><\/h2>\n<\/dt>\n<dd>\n<table class=\"responsive\">\n<colgroup>\n<col width=\"20%\">\n<col>\n<col>\n      <\/colgroup>\n<tbody>\n<tr>\n<td><b>User-Agent in HTTP requests<\/b><\/td>\n<td>\n            <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td><b>robots.txt<\/b><\/td>\n<td>\n<table class=\"nested fixed vertical-rules\">\n<tr>\n<td width=\"30%\">User-agent token in robots.txt<\/td>\n<td>\n<p><code>APIs-Google<\/code><\/p>\n<hr>\n<p>The global user agent (<code>*<\/code>) is ignored.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>Example robots.txt group<\/td>\n<td>\n                  <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/tr>\n<tr>\n<td><b>Affected products<\/b><\/td>\n<td>\n            Crawling preferences addressed to the <code>APIs-Google<\/code> user agent affect the<br \/>\n            delivery of push notification messages by Google APIs.\n          <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/dd>\n<dt>\n<h2 class=\"add-link\" id=\"adsbot-mobile-web\" tabindex=\"-1\">AdsBot Mobile Web<\/h2>\n<\/dt>\n<dd>\n<table class=\"responsive\">\n<colgroup>\n<col width=\"20%\">\n<col>\n<col>\n      <\/colgroup>\n<tbody>\n<tr>\n<td><b>User-Agent in HTTP requests<\/b><\/td>\n<td>\n            <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td><b>robots.txt<\/b><\/td>\n<td>\n<table class=\"nested fixed vertical-rules\">\n<tr>\n<td width=\"30%\">User-agent token in robots.txt<\/td>\n<td>\n<p><code>AdsBot-Google-Mobile<\/code><\/p>\n<hr>\n<p>The global user agent (<code>*<\/code>) is ignored.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>Example robots.txt group<\/td>\n<td>\n                  <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/tr>\n<tr>\n<td><b>Affected products<\/b><\/td>\n<td>\n            Crawling preferences addressed to the <code>AdsBot-Google-Mobile<\/code> user agent<br \/>\n            affect Google Ads&#8217; ability to check<br \/>\n            <a href=\"https:\/\/support.google.com\/google-ads\/answer\/2404197\" class=\"external-link\">web page ad quality<\/a>.\n          <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/dd>\n<dt>\n<h2 class=\"add-link\" id=\"adsbot\" tabindex=\"-1\">AdsBot<\/h2>\n<\/dt>\n<dd>\n<table class=\"responsive\">\n<colgroup>\n<col width=\"20%\">\n<col>\n<col>\n      <\/colgroup>\n<tbody>\n<tr>\n<td><b>User-Agent in HTTP requests<\/b><\/td>\n<td>\n            <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td><b>robots.txt<\/b><\/td>\n<td>\n<table class=\"nested fixed vertical-rules\">\n<tr>\n<td width=\"30%\">User-agent token in robots.txt<\/td>\n<td>\n<p><code>AdsBot-Google<\/code><\/p>\n<hr>\n<p>The global user agent (<code>*<\/code>) is ignored.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>Example robots.txt group<\/td>\n<td>\n                  <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/tr>\n<tr>\n<td><b>Affected products<\/b><\/td>\n<td>\n            Crawling preferences addressed to the <code>AdsBot-Google<\/code> user agent affect<br \/>\n            Google Ads&#8217; ability to check<br \/>\n            <a href=\"https:\/\/support.google.com\/google-ads\/answer\/2404197\" class=\"external-link\">web page ad quality<\/a>.\n          <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/dd>\n<dt>\n<h2 class=\"add-link\" id=\"adsense\" tabindex=\"-1\"><a href=\"https:\/\/support.google.com\/adsense\/answer\/99376\" class=\"external-link\">AdSense<\/a><\/h2>\n<\/dt>\n<dd>\n<table class=\"responsive\">\n<colgroup>\n<col width=\"20%\">\n<col>\n<col>\n      <\/colgroup>\n<tbody>\n<tr>\n<td><b>User-Agent in HTTP requests<\/b><\/td>\n<td>\n<table class=\"nested fixed vertical-rules\">\n<tr>\n<td width=\"30%\">Desktop agent<\/td>\n<td>\n                  <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td>Mobile agent<\/td>\n<td>\n                  <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/tr>\n<tr>\n<td><b>robots.txt<\/b><\/td>\n<td>\n<table class=\"nested fixed vertical-rules\">\n<tr>\n<td width=\"30%\">User-agent token in robots.txt<\/td>\n<td>\n                  <code>Mediapartners-Google<\/code><\/p>\n<hr>\n<p>The global user agent (<code>*<\/code>) is ignored.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>Example robots.txt group<\/td>\n<td>\n                  <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/tr>\n<tr>\n<td><b>Affected products<\/b><\/td>\n<td>\n            Crawling preferences addressed to the <code>Mediapartners-Google<\/code> user agent<br \/>\n            affect Google AdSense. The AdSense crawler visits participating sites in order to<br \/>\n            provide them with relevant ads.\n          <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/dd>\n<dt>\n<h2 class=\"add-link\" id=\"google-safety\" tabindex=\"-1\">Google-Safety<\/h2>\n<\/dt>\n<dd>\n<table class=\"responsive\">\n<colgroup>\n<col width=\"20%\">\n<col>\n<col>\n      <\/colgroup>\n<tbody>\n<tr>\n<td><b>User-Agent in HTTP requests<\/b><\/td>\n<td>\n            <span><\/span><\/p>\n<div><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td><b>robots.txt<\/b><\/td>\n<td>\n            The Google-Safety user agent ignores robots.txt rules.\n          <\/td>\n<\/tr>\n<tr>\n<td><b>Affected products<\/b><\/td>\n<td>\n            The Google-Safety user agent handles abuse-specific crawling, such as malware discovery<br \/>\n            for publicly posted links on Google properties. As such it&#8217;s unaffected by crawling<br \/>\n            preferences.\n          <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/dd>\n<\/dl>\n","protected":false},"excerpt":{"rendered":"<p>Google special-case crawlers are used by specific Google products where there&#8217;s an agreement between the crawled site and the product about the crawl process. This overview will help you understand the special-case Google crawlers including the AdsBot user agent.<\/p>\n","protected":false},"menu_order":7000,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[708],"ar_kb_etiket":[],"class_list":["post-25668","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-crawling-infrastructure-crawling-and-indexing"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25668","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25668\/revisions"}],"predecessor-version":[{"id":25689,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25668\/revisions\/25689"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27663"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25668"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25668"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25668"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}