The special-case crawlers are used by specific Google products where there’s an agreement between
the crawled site and the product about the crawl process. For example, AdsBot ignores
the global robots.txt user agent (*) with the ad publisher’s permission. The general
technical properties
of Google’s crawlers also apply to the special-case crawlers.
The
special-case crawlers may ignore robots.txt rules and so they operate from a different IP range
than the common crawlers. The IP ranges are published in the
special-crawlers.json object. The
special-case crawlers’ reverse DNS mask matches
rate-limited-proxy-***-***-***-***.google.com.
The following list shows the special-case crawlers, their user agent strings as they appear in
the HTTP requests, their user agent tokens for the User-agent: line in robots.txt,
and the products that are affected by crawl preferences for the crawler. The list is not
exhaustive, it only covers the requestors that are more likely to show up in log files and that
we’ve received questions about.
APIs-Google
User-Agent in HTTP requests robots.txt User-agent token in robots.txt APIs-GoogleThe global user agent (
*) is ignored.Example robots.txt group Affected products Crawling preferences addressed to the APIs-Googleuser agent affect the
delivery of push notification messages by Google APIs.AdsBot Mobile Web
User-Agent in HTTP requests robots.txt User-agent token in robots.txt AdsBot-Google-MobileThe global user agent (
*) is ignored.Example robots.txt group Affected products Crawling preferences addressed to the AdsBot-Google-Mobileuser agent
affect Google Ads’ ability to check
web page ad quality.AdsBot
User-Agent in HTTP requests robots.txt User-agent token in robots.txt AdsBot-GoogleThe global user agent (
*) is ignored.Example robots.txt group Affected products Crawling preferences addressed to the AdsBot-Googleuser agent affect
Google Ads’ ability to check
web page ad quality.AdSense
User-Agent in HTTP requests Desktop agent Mobile agent robots.txt User-agent token in robots.txt Mediapartners-GoogleThe global user agent (
*) is ignored.Example robots.txt group Affected products Crawling preferences addressed to the Mediapartners-Googleuser agent
affect Google AdSense. The AdSense crawler visits participating sites in order to
provide them with relevant ads.Google-Safety
User-Agent in HTTP requests robots.txt The Google-Safety user agent ignores robots.txt rules. Affected products The Google-Safety user agent handles abuse-specific crawling, such as malware discovery
for publicly posted links on Google properties. As such it’s unaffected by crawling
preferences.
Cómo lo aplicamos
Special-case crawlers ignore some of the usual rules, which surprises teams that assume robots.txt covers everything. We check for them specifically when a client reports unexpected fetch traffic.
Servicios relacionados