Crawling infrastructureAPIs-Google user agent
APIs-Google is the user agent used by Google APIs to deliver push notifications. Explore this guide to learn how APIs-Google…
Crawling infrastructure
Crawling infrastructureAuthenticate requests with Web Bot Auth (experimental)
Learn what Web Bot Auth is, the current status of Google's implementation, and how you can implement verification during Google's…
Crawling infrastructureChangelog
Track major updates to Google's crawling documentation, including changes to Google's list of user agents, crawlers, and user-triggered fetchers.
Crawling infrastructureDebug network and DNS errors for Google’s crawlers
See how different network and DNS errors can affect Google's crawlers and how to debug network and DNS errors.
Crawling infrastructureFeedfetcher
Feedfetcher is how Google grabs site feeds. Explore this guide to learn what Google Feedfetcher is and how this user-controlled…
Crawling infrastructureGoogle Read Aloud user agent
Google-Read-Aloud is the user agent for the Google Read Aloud service. Learn more about how this service can work for…
Crawling infrastructureHow Google interprets the robots.txt specification
Learn specific details about the different robots.txt file rules and how Google interprets the robots.txt specification.
Crawling infrastructureHow HTTP status codes affect Google’s crawlers
See how different HTTP status codes can affect Google's crawlers.
Crawling infrastructureHow to write and submit a robots.txt file
A robots.txt file lives at the root of your site. Learn how to create a robots.txt file, see examples, and…
Crawling infrastructureList of Google user-triggered fetchers
Google user-triggered fetchers perform actions on behalf of users. This overview will help you understand the user-triggered fetchers Google uses.
Crawling infrastructureList of Google’s common crawlers
Google crawlers discover and scan websites. This overview will help you understand the common Google crawlers including the Googlebot user…
Crawling infrastructureList of Google’s special-case crawlers
Google special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product…
Crawling infrastructureManaging crawling of faceted navigation URLs
Learn what faceted navigation is and how you can optimize Google's crawling of large number of faceted navigation URLs.
Crawling infrastructureOptimize your crawl budget
Learn what crawl budget is and how you can optimize Google's crawling of large and frequently updated websites.
Overview of Google crawlers and fetchers (user agents)
Understand the technical properties of Google crawlers and fetchers, including supported transfer protocols, caching, and file size limits.
Crawling infrastructureReduce the Google crawl rate
If Google's crawl rate overwhelms your server, explore this document to learn how to reduce Google's crawl rate and stop…
Things to know about Google’s web crawling
Learn about how Google crawls the web.
Crawling infrastructureUpdate your robots.txt file
With the robots.txt report, you can check whether Google can process your robots.txt files. Follow these steps to submit updated…
Crawling infrastructureVerify requests from Google crawlers and fetchers
You can check if a web crawler really is Googlebot (or another Google user agent). Follow these steps to verify…