Page and Content Metadata

The Robots Meta Tag and X-Robots-Tag

Where robots.txt controls crawling for the whole site, the robots meta tag controls indexing and serving for a single page. It comes in two forms: an HTML <meta> element, and an X-Robots-Tag HTTP header for non-HTML files.

The main directives

  • noindex, do not show this page in results.
  • nofollow, do not follow links on this page.
  • none, equivalent to noindex, nofollow.
  • nosnippet, no text snippet.
  • max-snippet:[n], max-image-preview:[setting], max-video-preview:[n].
  • noarchive, notranslate, noimageindex.
  • unavailable_after:[date], drop from the index after a date.

The combination that fails

Blocking a URL in robots.txt and adding noindex to the page. Google never fetches the page, never reads the tag, and the URL can remain indexed. To remove a page: allow crawling, add noindex, wait for a recrawl, and only then block it if you want to.

Alien Road recommends: use data-nosnippet on specific elements rather than nosnippet on the whole page when you only want to keep prices or personal details out of a snippet. Suppressing the entire snippet reliably reduces click-through; suppressing one paragraph does not.

, ,
Поделиться

© Copyright 2026 Alien Road. All rights reserved.