February 10, 2006<\/p>\n
\n Earlier this week, we told you about a feature we made available through the Sitemaps program that \n The Sitemaps robots.txt tool reads the robots.txt file in the same way Googlebot does. If the tool \n This tool provides results only for Google user-agents (such as Googlebot). Other bots may not \n A robots.txt file is valid only when it’s located in the root of a site. So, if you are looking at \n If you test a URL that resolves to a file (such as Consider this robots.txt file:<\/p>\n \n If you test \n You might see a “syntax not understood” error for a few different reasons. The most common one is \n We are working on a few known issues with the tool, including the way the tool processes
\n analyzes the robots.txt file for a site.<\/a>
\n Here are more details about that feature.\n<\/p>\nWhat the analysis means<\/h2>\n
\n interprets a line as a syntax error, Googlebot doesn’t understand that line. If the tool shows
\n that a URL is allowed, Googlebot interprets that URL as allowed.\n<\/p>\n
\n interpret the robots.txt file in the same way. For instance, Googlebot supports an extended
\n definition of the standard. It understands Allow:<\/code> lines, as well as
\n *<\/code> and $<\/code>. So while the tool shows lines that include these extensions as
\n understood, remember that this applies only to Googlebot and not necessarily to other bots that
\n may crawl your site.\n<\/p>\nSubdirectory sites<\/h2>\n
\n a site in your account that is located in a subdirectory (such as
\n https:\/\/www.example.com\/mysite\/<\/code>), we show you information on the robots.txt file at
\n the root (https:\/\/www.example.com\/robots.txt<\/code>). You may not have access to this file,
\n but we show it to you because the robots.txt file can impact crawling of your subdirectory site
\n and you may want to make sure it’s allowing URLs as you expect.\n<\/p>\nTesting access to directories<\/h2>\n
\n https:\/\/www.example.com\/myfile.html<\/code>), this tool can determine if the robots.txt file
\n allows or blocks that file. If you test a URL that resolves to a directory (such as
\n https:\/\/www.example.com\/folder1\/<\/code>), this tool can determine if the robots.txt file
\n allows or blocks access to that URL, but it can’t tell you about access to the files inside that
\n folder. The robots.txt file may have set restrictions on URLs inside the folder that are different
\n than the URL of the folder itself.\n<\/p>\nhttps:\/\/www.example.com\/folder1\/<\/code>, the tool will say that it’s blocked.
\n But if you test https:\/\/www.example.com\/folder1\/myfile.html<\/code>, you’ll see that it’s not
\n blocked even though it’s located inside of folder1<\/code>.\n<\/p>\nSyntax not understood<\/h2>\n
\n that Googlebot couldn’t parse the line. However, some other potential reasons are:\n<\/p>\n\n
200<\/code> for
\n pages that aren’t found. If the server is configured this way, then when Googlebot requests the
\n robots.txt file, the server returns a page. However, this page isn’t actually a robots.txt
\n file, so Googlebot can’t process it.\n <\/li>\n
\n receives a different type of file (for instance, an HTML file), this tool won’t show a syntax
\n error for every line in the file. Rather, it shows one error for the entire file.\n <\/li>\n
\n other than the
\n robots.txt standard<\/a>.
\n If Googlebot encounters one of the more common additional rules, the tool lists them syntax
\n errors.\n <\/li>\n<\/ul>\nKnown issues<\/h2>\n
\n capitalization and the analysis with Google user-agents other than Googlebot. We’ll keep you
\n posted as we get these issues resolved.\n<\/p>\n