Guidelines for sites with explicit content
If your site hosts explicit or adult content, you can help Google understand the nature of your site and content,…
Explicit contentWhat to do if your site is incorrectly flagged as explicit in…
Learn how to determine if your site is being incorrectly flagged as explicit in Google Search results, and how to…
Crawl Budget: When It Matters and When It Does Not
Crawl rate and crawl demand, why most sites can ignore both, and what to do when yours cannot.
Crawling and indexingDiagnosing Crawl Errors
Reading what Search Console reports, and separating errors that matter from ones that are working as intended.
DocumentationGet your website on Google
Check to see if your site is on Google and learn how to make your content more visible in Google…
Crawling and indexingGoogle Crawlers and How to Verify Them
Which user agents Google operates, and how to confirm a request claiming to be Googlebot really is.
Crawling and indexingHow Google Interprets robots.txt
Matching rules, wildcards, precedence and the size limit, the details that decide whether your file does what you think.
Ranking and search appearanceImage Metadata, Licensing and Attribution
IPTC metadata, licensable image results and keeping credit attached to your work.
Crawling and indexingMobile-First Indexing
Google indexes the mobile version. What that means for content parity and markup.
SEO fundamentalsOptimising for AI Features in Search
What actually changes when answers are generated rather than listed, and what does not change at all.
Reading Third-Party SEO Tool Scores Correctly
Why tool metrics are estimates, and how to use them without being misled.
Crawling and indexingRemoving Outdated Content You Do Not Control
When the page has changed or gone but Google still shows the old version.
Ranking and search appearanceSchema Markup Testing Tool
Use the Rich Result Test to see what Google results can be generated for your pages and the schema markup…
Monitoring and debuggingSearch Operators for Diagnosis
Using Google itself to inspect what is indexed, and reading the results correctly.
Site-specific guidesSensitive Content and SafeSearch
How Google classifies adult content, and what to do about incorrect flags.
Web Stories
A full-screen tappable format with its own requirements and its own policy set.
Crawling infrastructureAPIs-Google user agent
APIs-Google is the user agent used by Google APIs to deliver push notifications. Explore this guide to learn how APIs-Google…
Crawling infrastructureAuthenticate requests with Web Bot Auth (experimental)
Learn what Web Bot Auth is, the current status of Google's implementation, and how you can implement verification during Google's…
Crawling infrastructureChangelog
Track major updates to Google's crawling documentation, including changes to Google's list of user agents, crawlers, and user-triggered fetchers.
Ranking and search appearanceCore Updates and How to Respond
What a core update is, why "fixing" it usually fails, and what actually helps.
Crawling infrastructureDebug network and DNS errors for Google’s crawlers
See how different network and DNS errors can affect Google's crawlers and how to debug network and DNS errors.
Crawling infrastructureFeedfetcher
Feedfetcher is how Google grabs site feeds. Explore this guide to learn what Google Feedfetcher is and how this user-controlled…
Crawling infrastructureGoogle Read Aloud user agent
Google-Read-Aloud is the user agent for the Google Read Aloud service. Learn more about how this service can work for…
Crawling infrastructureHow Google interprets the robots.txt specification
Learn specific details about the different robots.txt file rules and how Google interprets the robots.txt specification.
Crawling infrastructureHow HTTP status codes affect Google’s crawlers
See how different HTTP status codes can affect Google's crawlers.
Crawling infrastructureHow to write and submit a robots.txt file
A robots.txt file lives at the root of your site. Learn how to create a robots.txt file, see examples, and…
Crawling infrastructureList of Google user-triggered fetchers
Google user-triggered fetchers perform actions on behalf of users. This overview will help you understand the user-triggered fetchers Google uses.
Crawling infrastructureList of Google’s common crawlers
Google crawlers discover and scan websites. This overview will help you understand the common Google crawlers including the Googlebot user…
Crawling infrastructureList of Google’s special-case crawlers
Google special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product…
Crawling infrastructureManaging crawling of faceted navigation URLs
Learn what faceted navigation is and how you can optimize Google's crawling of large number of faceted navigation URLs.
Crawling infrastructureOptimize your crawl budget
Learn what crawl budget is and how you can optimize Google's crawling of large and frequently updated websites.
Overview of Google crawlers and fetchers (user agents)
Understand the technical properties of Google crawlers and fetchers, including supported transfer protocols, caching, and file size limits.
Crawling infrastructureReduce the Google crawl rate
If Google's crawl rate overwhelms your server, explore this document to learn how to reduce Google's crawl rate and stop…
Things to know about Google’s web crawling
Learn about how Google crawls the web.
Crawling infrastructureUpdate your robots.txt file
With the robots.txt report, you can check whether Google can process your robots.txt files. Follow these steps to submit updated…
Crawling infrastructureVerify requests from Google crawlers and fetchers
You can check if a web crawler really is Googlebot (or another Google user agent). Follow these steps to verify…