Tuesday, July 02, 2019<\/p>\n
\n Yesterday we announced that we’re \n The internet draft<\/a> we published yesterday provides an \n While open-sourcing our parser library, we analyzed the usage of robots.txt rules. In particular, \n In the interest of maintaining a healthy ecosystem and preparing for potential future open source \n For more guidance about how to remove information from Google’s search results, visit our
\n open-sourcing Google’s production robots.txt parser<\/a>.
\n It was an exciting moment that paves the road for potential Search open sourcing projects in the
\n future! Feedback is helpful, and we’re eagerly collecting questions from
\n developers<\/a> and
\n webmasters<\/a> alike. One question
\n stood out, which we’ll address in this post:
\n
\n Why isn’t a code handler for other rules like crawl-delay included in the code?\n <\/p>\n
\n extensible architecture for rules that are not part of the standard. This means that if a
\n crawler wanted to support their own line like unicorns: allowed<\/code>,
\n they could. To demonstrate how this would look in a parser, we included a very common line,
\n sitemap, in our open-source robots.txt parser<\/a>.\n <\/p>\n
\n we focused on rules unsupported by the internet draft, such as
\n crawl-delay<\/code>, nofollow<\/code>, and
\n noindex<\/code>. Since these rules were never documented by Google,
\n naturally, their usage in relation to Googlebot is very low. Digging further, we saw their usage
\n was contradicted by other rules in all but 0.001% of all robots.txt files on the internet.
\n These mistakes hurt websites’ presence in Google’s search results in ways we don’t think
\n webmasters intended.\n <\/p>\n
\n releases, we’re retiring all code that handles unsupported and unpublished rules (such as
\n noindex<\/code>) on September 1, 2019. For those of you who relied on the
\n noindex<\/code> indexing rule in the
\n robots.txt<\/code> file, which controls crawling, there are a number of
\n alternative options:\n <\/p>\n\n
noindex<\/code><\/a>
\n in robots<\/span> meta<\/code> tags:<\/b> Supported both in the HTTP response headers and in HTML, the
\n noindex<\/code> rule is the most effective way to remove URLs from
\n the index when crawling is allowed.\n <\/li>\n404<\/code> and 410<\/code> HTTP status codes<\/a>:<\/b>
\n Both status codes mean that the page does not exist, which will drop such URLs from Google’s
\n index once they’re crawled and processed.\n <\/li>\n
\n subscription or paywalled content<\/a>,
\n hiding a page behind a login will generally remove it from Google’s index.\n <\/li>\nDisallow<\/code> in robots.txt<\/code>:<\/b> Search engines can only index pages
\n that they know about, so blocking the page from being crawled usually means its content won’t
\n be indexed. While the search engine may also index a URL based on links from other pages,
\n without seeing the content itself, we aim to make such pages less visible in the future.\n <\/li>\n
\n The tool is a quick and easy method to remove a URL temporarily from Google’s search results.\n <\/li>\n<\/ul>\n
\n Help Center<\/a>.
\n If you have questions, you can find us on Twitter<\/a>
\n and in our Webmaster Community<\/a>,
\n both offline<\/a> and online.\n <\/p>\n