robots.txt tells crawlers which URLs they may request. It does not keep a page out of the index. A blocked URL that other pages link to can still appear in results, usually with no description, because Google was not allowed to read it.
The trap
If you want a page out of Search, use noindex. But noindex only works if Google can fetch the page and read the tag. Blocking the URL in robots.txt prevents Google from seeing the noindex. The two directives cancel each other out, and the page stays indexed.
Basic rules
- Must live at the root of the host:
example.com/robots.txt. - Applies per host and protocol,
httpandhttpsare separate. - The most specific matching rule wins, not the first.
- A missing file (404) means everything is allowed. A 5xx means Google may stop crawling entirely.
Alien Road recommends: never block CSS or JavaScript. Google renders pages before indexing them; blocked resources produce a broken render, and Google indexes what it rendered. We still find this rule in inherited files, usually copied from a template written years ago, and removing it has produced immediate coverage improvements.