robots.txt
robots.txt: What It Controls and What It Does Not
The single most misunderstood file in SEO. It controls crawling, not indexing, and the difference matters.
- All
- Getting Started 4
- Search Essentials 3
- Discover and Other Surfaces 3
- Structured Data 3
- URL Structure and Links 3
- Site Moves and Changes 3
- Page and Content Metadata 3
- Canonicalization 3
- Sitemaps 3 Crawler Management 3 robots.txt 2 JavaScript SEO 2 Ecommerce 2 Removals 2 Generative AI in Search 2 International and Multilingual 2 Preventing Abuse 2 How Search Works 2 Title Links and Snippets 2 Images in Search 2 Search Console 2 Page Experience 2 Ranking Systems and Updates 2 Sensitive and Explicit Content 1 Debugging Traffic Drops 1 Search Operators 1 Site Identity in Results 1 Videos in Search 1 Mobile-First Indexing 1 Helpful, People-First Content 1 Monitoring and Debugging 0 Ranking and Search Appearance 0 Site-Specific Guides 0 Crawling and Indexing 0 SEO Fundamentals 0