Crawler Management
Google Crawlers and How to Verify Them
Which user agents Google operates, and how to confirm a request claiming to be Googlebot really is.
- All
- Getting Started 4
- Search Essentials 3
- Discover and Other Surfaces 3
- Structured Data 3
- URL Structure and Links 3
- Site Moves and Changes 3
- Page and Content Metadata 3
- Canonicalization 3
- Sitemaps 3 Crawler Management 3 robots.txt 2 JavaScript SEO 2 Ecommerce 2 Removals 2 Generative AI in Search 2 International and Multilingual 2 Preventing Abuse 2 How Search Works 2 Title Links and Snippets 2 Images in Search 2 Search Console 2 Page Experience 2 Ranking Systems and Updates 2 Sensitive and Explicit Content 1 Debugging Traffic Drops 1 Search Operators 1 Site Identity in Results 1 Videos in Search 1 Mobile-First Indexing 1 Helpful, People-First Content 1 Monitoring and Debugging 0 Ranking and Search Appearance 0 Site-Specific Guides 0 Crawling and Indexing 0 SEO Fundamentals 0