Friday, March 14, 2025<\/p>\n
\n With the robots.txt file<\/a>, site owners \n Meta \n To give the same level of control to non-HTML content, the “ \n The syntax is simple and extensible. The rules are generally either implemented by the web \n For example, the following rules tell all crawlers not to use the associated page for indexing:\n<\/p>\n \n Looking at existing \n You can check the HTTP response headers for individual URLs with Chrome’s developer tools, in the \n Other examples of what you can do:\n<\/p>\n Don’t show a snippet for this page or document.<\/p>\n<\/td>\n or in HTML:<\/p>\n Don’t index this page in These controls explicitly specify one crawler.<\/p>\n<\/td>\n oder<\/p>\n Note that the most restrictive, valid directives apply, so for oder<\/p>\n \n How do you choose which one to use? Fundamentally robots.txt and page-level controls are similar, \n All of these controls are extensible by nature. Over the years, site-owners, crawler operators, \n Under the REP umbrella, site owners have control over what is crawled and how the crawled data
\n have a simple way to control which parts of a website are
\n accessible by crawlers.
\n To help site owners further express how search engines and web
\n crawlers can use their pages, the community involved in developing web standards
\n came
\n up with robots meta<\/code> tags in 1996<\/a>,
\n just a few months after meta<\/code> tags were proposed for HTML (and anecdotally,
\n also before Google
\n was founded). Later,
\n X-Robots-Tag<\/code>
\n HTTP response headers<\/a> were added.
\n These instructions are sent together with a URL, so crawlers can only take them into account
\n if they’re not disallowed from crawling the URL through the robots.txt file. Together, they
\n form the Robots Exclusion Protocol (REP).\n<\/p>\nA look at robots
meta<\/code> tags<\/h2>\n
\n tags (or elements)<\/a> are a way to include machine-readable metadata.
\n Robots meta<\/code> tags are one “kind” of meta<\/code> tag, and apply to crawlers, including search engine
\n crawlers. They signal: Is the content blocked from indexing? Should links on the page not be followed for
\n crawling? It’s easy to give this information on the page directly with robots meta<\/code> tags.\n<\/p>\nA Robots Exclusion Protocol for any URL<\/h2>\n
X-Robots-Tag<\/code>” HTTP response header
\n was created. These
\n HTTP headers<\/a>
\n are considered a part of the REP as well.
\n The header supports the same values as the robots meta<\/code> tag,
\n and can be added to any piece of content served online.
\n Besides HTML, Google supports it for content like PDFs, document files, and even images.
\n Most of these file formats don’t have a mechanism equivalent to meta<\/code> tags, so a HTTP
\n response header is helpful.\n<\/p>\nGetting started with robots
meta<\/code> tags and headers<\/h2>\n
\n developer or through a Content Management System (CMS), where site-owners may have checkboxes
\n or drop-down menus to select their preferences.
\n These controls can address a specific crawler, such as Googlebot or, by omitting a specific name,
\n address all crawlers that support these values.\n<\/p>\n\n
meta<\/code> tag, on a web page:\nmeta<\/code> tags or response headers is a bit more involved,
\n and requires direct examination of page content or headers.
\n You can view HTML
\n meta<\/code> tags on any page either looking at the page source in your browser, or using Chrome’s
\n developer tools to
\n inspect
\n the page<\/a>. \n <\/p>\n<\/li>\n
\n response header<\/a>:<\/p>\n
\n network panel<\/a>.\n <\/p>\n<\/li>\n<\/ul>\n\n
\n \n \n In HTTP header:<\/p>\n \n \n ExampleBot-News<\/code>, without specifying a preference for others.<\/p>\n\n \n \n ExampleBot<\/code> should not show a snippet, and additionally, all crawlers should not follow the
\n links on this page.<\/p>\nExampleBot<\/code> the directive
\n would be combined as “nosnippet, nofollow<\/code>“.<\/p>\n<\/td>\n\n Choosing a REP mechanism<\/h2>\n
\n but not completely interchangeable.
\n Sometimes there’s a specific action that’s only possible
\n with one of the mechanisms, for example,
\n if it’s desired to stop the act of crawling (such as
\n for endless search results pages, possible with robots.txt),
\n if you need a control for an FTP server (possible with robots.txt),
\n or if it’s desired not to have a snippet shown for a page (which is only possible with page-level
\n elements).
\n If you don’t need to differentiate between blocking crawling and blocking indexing,
\n one approach is to use robots.txt for broader controls (to block large parts of a website),
\n and page-level controls for blocking individual pages.\n<\/p>\nRobots Exclusion Protocol\u2014a powerful, living standard<\/h2>\n
\n and search engines have worked together to evolve them.
\n Historically it started with a handful
\n of values, including noindex<\/code> and nofollow<\/code>, then later on more values like
\n nosnippet<\/code>, noarchive<\/code>,
\n and max-snippet:<\/code> were adopted.
\n And sometimes values are deprecated, as was the case with noodp<\/code>,
\n which used snippets from
\n DMOZ \/ Open Directory Project<\/a>
\n before the directory was closed.
\n There’s a
\n plethora of values<\/a> supported
\n by Google for site owners, and a similar amount from other large crawler operators.\n<\/p>\n
\n is used in search engines.
\n They can do this on a broad level for bigger parts of websites,
\n or at a very granular level, for individual pages, even for images within pages.
\n These controls are well-known, available in all common content management systems,
\n broadly supported by commercial operators, and used on
\n billions of hosts on the internet today.\n<\/p>\n