Crawling infrastructure
Analyzing Googlebot Support and Benefits for HTTP/2 and HTTP/3
Googlebot actively supports HTTP/2 and HTTP/3 to improve crawling efficiency. This guide details how these protocols optimize data transfer for…
Crawling infrastructure
Analyzing Googlebot’s Mobile-First Indexing Crawl Patterns
Mobile-first indexing means Google uses the smartphone version of your content as the primary source for indexing. Learn how to…
Crawling infrastructure
Detecting and Resolving Soft 404 Errors During Crawl Sessions
Learn how soft 404 errors impact your site crawl sessions and how to fix them efficiently using Google Search Central…
Crawling infrastructure
Detecting Crawler Traps in Dynamic URL Structures and Facets
Crawler trap detection identifies infinite URL loops caused by dynamic parameters to protect your crawl budget and improve indexing efficiency.
Crawling infrastructure
Diagnosing Googlebot TLS Handshake Failure and Cipher Issues
A Googlebot TLS handshake failure occurs when your server cannot establish a secure connection, preventing Google from indexing your content…
Crawling infrastructure
Handling Googlebot Crawl Spikes: Identifying and Blocking Bad Bots
Discover how to identify genuine Googlebot activity, manage sudden server crawl spikes, and block malicious traffic while maintaining search performance.
Crawling infrastructure
Handling JavaScript-Heavy Content with Google’s Web Rendering Service
Google's Web Rendering Service processes JavaScript to index dynamic content. Learn how to optimize your site for efficient crawling and…
Crawling infrastructure
Impact of Hreflang Tags on Googlebot Crawl Discovery and Frequency
Hreflang tags guide Googlebot to relevant regional content, optimizing crawl efficiency by clarifying site structure for specific language audiences. Effective…
Crawling infrastructure
Impact of Server-Side Rendering vs Client-Side on Crawl Efficiency
Server-side rendering delivers fully formed HTML to search bots immediately, whereas client-side rendering requires executing JavaScript to discover page content…
Crawling infrastructure
Mitigating Excessive Crawling of High-Cardinality URL Parameters
High-cardinality parameters generate millions of low-value URLs that exhaust Googlebot crawl resources. Implementing robots.txt rules and clear URL patterns resolves…
Crawling infrastructure
Optimizing Conditional Crawling via HTTP 304 Not Modified Headers
Learn to optimize server responses using HTTP 304 headers to save crawl budget and reduce server load for Googlebot efficiently.…
Crawling infrastructure
Troubleshooting Googlebot Crawl Timeouts and Connectivity Issues
Address Googlebot crawl timeout errors by identifying server-side bottlenecks. Ensure your content remains indexed by optimizing server response times.
Crawling infrastructureAPIs-Google user agent
APIs-Google is the user agent used by Google APIs to deliver push notifications. Explore this guide to learn how APIs-Google…
Crawling infrastructureAuthenticate requests with Web Bot Auth (experimental)
Learn what Web Bot Auth is, the current status of Google's implementation, and how you can implement verification during Google's…
Crawling infrastructureChangelog
Track major updates to Google's crawling documentation, including changes to Google's list of user agents, crawlers, and user-triggered fetchers.
Crawling infrastructureDebug network and DNS errors for Google’s crawlers
See how different network and DNS errors can affect Google's crawlers and how to debug network and DNS errors.
Crawling infrastructureFeedfetcher
Feedfetcher is how Google grabs site feeds. Explore this guide to learn what Google Feedfetcher is and how this user-controlled…
Crawling infrastructureGoogle Read Aloud user agent
Google-Read-Aloud is the user agent for the Google Read Aloud service. Learn more about how this service can work for…
Crawling infrastructureHow Google interprets the robots.txt specification
Learn specific details about the different robots.txt file rules and how Google interprets the robots.txt specification.
Crawling infrastructureHow HTTP status codes affect Google’s crawlers
See how different HTTP status codes can affect Google's crawlers.
Crawling infrastructureHow to write and submit a robots.txt file
A robots.txt file lives at the root of your site. Learn how to create a robots.txt file, see examples, and…
Crawling infrastructureList of Google user-triggered fetchers
Google user-triggered fetchers perform actions on behalf of users. This overview will help you understand the user-triggered fetchers Google uses.
Crawling infrastructureList of Google’s common crawlers
Google crawlers discover and scan websites. This overview will help you understand the common Google crawlers including the Googlebot user…
Crawling infrastructureList of Google’s special-case crawlers
Google special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product…
Crawling infrastructureManaging crawling of faceted navigation URLs
Learn what faceted navigation is and how you can optimize Google's crawling of large number of faceted navigation URLs.
Crawling infrastructureOptimize your crawl budget
Learn what crawl budget is and how you can optimize Google's crawling of large and frequently updated websites.
Overview of Google crawlers and fetchers (user agents)
Understand the technical properties of Google crawlers and fetchers, including supported transfer protocols, caching, and file size limits.
Crawling infrastructureReduce the Google crawl rate
If Google's crawl rate overwhelms your server, explore this document to learn how to reduce Google's crawl rate and stop…
Things to know about Google’s web crawling
Learn about how Google crawls the web.
Crawling infrastructureUpdate your robots.txt file
With the robots.txt report, you can check whether Google can process your robots.txt files. Follow these steps to submit updated…
Crawling infrastructureVerify requests from Google crawlers and fetchers
You can check if a web crawler really is Googlebot (or another Google user agent). Follow these steps to verify…