Monday, September 21, 2020<\/p>\n
\n Last year we released the \n With the intern season ending here at Google, we wanted to highlight two new releases related to \n First, we are releasing a \n Second, we are releasing an official \n As usual, we welcome your contributions to these projects. If you built something with the \n It was our genuine pleasure to host Andreea and Ian, and we’re sad that their internship is \n
\n robots.txt parser and matcher<\/a> that we use in
\n our production systems to the open source world. Since then, we’ve seen people build new tools
\n with it,
\n contribute<\/a> to the
\n open source library (effectively improving our production systems- thanks!), and release new
\n language versions like golang<\/a> and
\n rust<\/a>, which make it easier for
\n developers to build new tools.\n <\/p>\n
\n robots.txt that were made possible by two interns working on the Search Open Sourcing team,
\n Andreea Dutulescu<\/a> and
\n Ian Dolzhanskii<\/a>.<\/p>\nRobots.txt Specification Test<\/h2>\n
\n testing framework<\/a> for robots.txt
\n parser developers, created by Andreea. The project provides a testing tool that can validate
\n whether a robots.txt parser follows the Robots Exclusion Protocol, or to what extent. Currently
\n there is no official and thorough way to assess the correctness of a parser, so Andreea built a
\n tool that can be used to create robots.txt parsers that are following the protocol.\n <\/p>\nJava robots.txt parser and matcher<\/h2>\n
\n Java port of the C++ robots.txt parser<\/a>,
\n created by Ian. Java is the
\n 3rd most popular programming language<\/a>
\n on GitHub and it’s extensively used at Google as well, so no wonder it’s been the most requested
\n language port. The parser is a 1-to-1 translation of the C++ parser in terms of functions and
\n behavior, and it’s been thoroughly tested for parity against a large corpora of robots.txt
\n rules. Teams are already planning to use the Java robots.txt parser in Google production
\n systems, and we hope that you’ll find it useful, too.\n <\/p>\n
\n C++ robots.txt parser<\/a> or with these new
\n releases, let us know so we can potentially help you spread the word! If you found a bug, help
\n us fix it by opening an issue on GitHub or directly contributing with a pull request. If you
\n have questions or comments about these projects, catch us on
\n Twitter<\/a>!\n <\/p>\n
\n ending. Their contributions help make the Internet a better place and we hope that we can
\n welcome them back to Google in the future.\n <\/p>\n