Friday, March 28, 2025<\/p>\n
\n In the previous posts about the Robots Exclusion Protocol (REP), we explored what you can already
\n do with its various components — namely robots.txt and the URI level controls.
\n In this post we will explore how the REP can play a supporting role in the ever-evolving relation
\n between automatic clients and the human web.\n<\/p>\n
\n The REP — specifically robots.txt — became a standard in 2022 as \n That doesn’t mean that there are no other rules; any crawler operator can come up with their own \n Because the REP can in fact get “updates”. It’s a widely supported protocol and it should grow \n Due to its simplicity and wide adoption, the REP is an excellent candidate for carrying new \n The same goes for the REP URI level extensions, the \n The most important thing you, the reader, can do is to talk about your idea publicly and gather \n Similarly, if the protocol is lacking something, talk about it publicly. If your driver is to serve the common good, it’s worth it.<\/p>\n
\n RFC9309<\/a>.
\n However, the heavy lifting was done prior to its standardization: it was the test of time between
\n 1994 and 2022 that made it popular enough to be adopted by billions of hosts and virtually all
\n major crawler operators (excluding adversarial crawlers such as malware scanners). It is a
\n straightforward and elegant solution to express preferences with a simple yet versatile syntax.
\n In its 25 years of existence it barely had to evolve from its original form, it only got an
\n allow<\/code> rule if we only consider the rules that are universally supported by crawlers.\n<\/p>\n
\n rules. For example, rules like “clean-param<\/code>” and “crawl-delay<\/code>” are not
\n part of RFC9309, but they’re supported by some search engines — though not Google Search.
\n The “sitemap<\/code>” rule, which again is not part of RFC9309, is supported by all major
\n search engines. Given enough support, it could become an official rule in the REP.\n<\/p>\n
\n with the internet. Making changes to it is not impossible, but it’s not easy; it shouldn’t be
\n easy, exactly because the REP is widely supported. Like with any change to a standard, there has
\n to be a consensus that changes benefit the majority of the users of the protocol, both on the
\n publishers’ and the crawler operators’ side.\n<\/p>\n
\n crawling preferences: billions of publishers are already familiar with robots.txt and its syntax
\n for example, so making changes to it comes more naturally for them. On the flip side, crawler
\n operators already have robust, well tested parsers and matchers (and Google also open sourced its
\n own robots.txt parser<\/a>),
\n which means it’s highly likely that there won’t be parsing issues with new rules.\n<\/p>\nX-robots-tag<\/code> HTTP header and its
\n meta tag counterpart. If there is a need for a new rule to carry opt-out preferences, they’re
\n easily extensible. How though?\n<\/p>\n
\n supporters for that idea. Because the REP is a public standard, no one entity can make unilateral
\n changes to it; sure, they can implement support for something new on their side, but that won’t
\n become THE standard. But talking about that change and showing to the ecosystem — both
\n crawler operators and the publishing ecosystem — that it’s benefiting everyone will drive
\n consensus, and that paves the road to updating the standard.\n<\/p>\nsitemap<\/code>
\n became a widely supported rule in robots.txt because it was useful for content creators and search
\n engines alike, which paved the road to adoption of the extension. If you have a new idea for a
\n rule, ask the consumers of robots.txt and creators what they think about it and work with them to
\n hash out potential (and likely) issues they raise and write up a proposal.\n<\/p>\n