\n was only a de-facto standard. This had frustrating implications sometimes. On one hand, for
\n webmasters, it meant uncertainty in corner cases, like when their text editor included
\n BOM<\/a> characters in
\n their robots.txt files. On the other hand, for crawler and tool developers, it also brought
\n uncertainty; for example, how should they deal with robots.txt files that are hundreds of
\n megabytes large?\n<\/p>\n
![\"Googlebot](\"https:\/\/alienroad.com\/wp-content\/uploads\/kb-gorsel\/g-6bb39cb08b83.png\")
<\/p>\n
\n Today, we announced<\/a> that we’re spearheading the effort \n We’re here to help: we open sourced<\/a> \n We also included a testing tool in the open source package to help you test a few rules. Once \n \n If you want to check out the library, head over to our GitHub repository for the
\n to make the REP an internet standard. While this is an important step, it means extra work for
\n developers who parse robots.txt files.\n<\/p>\n
\n the C++ library that our production systems use for parsing and matching rules in robots.txt
\n files. This library has been around for 20 years and it contains pieces of code that were written
\n in the 90’s. Since then, the library evolved; we learned a lot about how webmasters write
\n robots.txt files and corner cases that we had to cover for, and added what we learned over the
\n years also to the internet draft when it made sense.\n<\/p>\n
\n built, the usage is very straightforward:\n<\/p>\nrobots_main <robots.txt content> <user_agent> <url><\/code>\n<\/p>\n
\n robots.txt parser<\/a>. We’d love
\n to see what you can build using it! If you built something using the library, drop us a comment on
\n Twitter<\/a>, and if you have comments
\n or questions about the library, find us on
\n GitHub<\/a>.\n<\/p>\n