crawlingHow Google Interprets robots.txt
Matching rules, wildcards, precedence and the size limit, the details that decide whether your file does what you think.
- All
- crawling 10
- indexing 9
- search console 5
- sitemap 4
- snippets 3
- images 3
- canonical 3
- robots.txt 3
- crawling 10 indexing 9 search console 5 sitemap 4 snippets 3 images 3 canonical 3 robots.txt 3 structured data 3 redirects 3 policy 3 mobile 3 penalty 2 video 2 traffic drop 2 links 2 recovery 2 javascript 2 rendering 2 debugging 2 noindex 2 migration 2 audit 2 spam 2 ranking 2 verification 2 schema 2 quality 2 removals 2 image SEO 1 IPTC 1 key moments 1 local business 1 core web vitals 1 metadata 1 CLS 1 performance 1 faq 1 title 1 SERP 1 licensing 1 LCP 1 meta description 1 featured snippet 1 alt text 1 INP 1 algorithm 1 reports 1 operators 1 site: 1 diagnosis 1 security 1 hacked 1 hreflang 1 localization 1 geotargeting 1 explicit 1 AMP 1 web stories 1 multilingual 1 systems 1 core update 1 safesearch 1 favicon 1 site name 1 sitelinks 1 discover 1 feed 1 translation 1 local 1 classification 1 product 1 search essentials 1 hiring 1 agency 1 maintenance 1 monitoring 1 housekeeping 1 tools 1 metrics 1 discovery 1 news 1 checklist 1 basics 1 crawl budget 1 large sites 1 content 1 E-E-A-T 1 AI content 1 AI features 1 AI Overviews 1 GEO 1 starter 1 googlebot 1 errors 1 testing 1 cloaking 1 URL 1 structure 1 anchor text 1 PDF 1 file types 1 responsive 1 rich results 1 301 1 cache 1 syntax 1 duplicate content 1 meta tags 1 html 1 robots 1 nofollow 1 sponsored 1 urgent 1 outdated 1 guidelines 1
crawling