February 24, 2006<\/p>\n
\n A couple of weeks ago, we \n A robots.txt file provides restrictions to search engine robots (known as “bots”) that crawl the \n Only if your site includes content that you don’t want search engines to index. If you want \n The robots.txt file must<\/em> reside in the root of the domain. A robots.txt file located in \n You can create this file in any text editor. It should be an The simplest robots.txt file uses two rules:<\/p>\n \n These two lines are considered a single entry in the file. You can include as many entries as you \n A user-agent is a specific search engine robot. The \n The \n URLs are case-sensitive. For instance, \n Google uses several user-agents. You can block access to any of them by including the bot name on \n Yes, Googlebot recognizes an extension to the robots.txt standard called \n You may want to use \n Those entries would block all pages inside the folder1 directory except for \n Yes, you can \n Yes, Googlebot interprets some pattern matching. This is an extension of the standard, so not all \n You can use an asterisk (
\n launched a robots.txt<\/a>
\n analysis tool. This tool gives you information about how
\n Googlebot<\/a>
\n interprets your robots.txt file. You can read more about the
\n robots.txt Robots Exclusion Standard<\/a>,
\n but we thought we’d answer some common questions here.\n<\/p>\nWhat is a robots.txt file?<\/h2>\n
\n web. These bots are automated, and before they access pages of a site, they check to see if a
\n robots.txt file exists that prevents them from accessing certain pages.\n<\/p>\nDoes my site need a robots.txt file?<\/h2>\n
\n search engines to index everything in your site, you don’t need a robots.txt file (not even an
\n empty one).\n<\/p>\nWhere should the robots.txt file be located?<\/h2>\n
\n a subdirectory isn’t valid, as bots only check for this file in the root of the domain. For
\n instance, https:\/\/www.example.com\/robots.txt<\/code> is a valid location. But,
\n https:\/\/www.example.com\/mysite\/robots.txt<\/code> is not. If you don’t have access to the
\n root of a domain, you can restrict access using the
\n Robots META tag.<\/a>\n<\/p>\nHow do I create a robots.txt file?<\/h2>\n
\n ASCII-encoded text file<\/a>,
\n not an HTML file. The filename should be lowercase.\n<\/p>\nWhat should the syntax of my robots.txt file be?<\/h2>\n
\n
User-Agent:<\/code><\/strong> the robot the following rule applies to<\/li>\nDisallow:<\/code><\/strong> the pages you want to block<\/li>\n<\/ul>\n
\n want. You can include multiple Disallow lines in one entry.\n<\/p>\nUser-Agent<\/code><\/h2>\n
\n Web Robots Database<\/a>
\n lists many common bots. You can set an entry to apply to a specific bot (by listing the name) or
\n you can set it to apply to all bots (by listing an asterisk). An entry that applies to all bots
\n looks like this:\n<\/p>\nDisallow<\/code><\/h2>\nDisallow<\/code> line lists the pages you want to block. You can list a specific URL or
\n a pattern. The entry should begin with a forward slash (\/).\n<\/p>\n\n
Disallow: \/<\/code>\n <\/li>\n
\n Disallow: \/private_directory\/<\/code>\n <\/li>\n
\n Disallow: \/private_file.html<\/code>\n <\/li>\n<\/ul>\nDisallow: \/private_file.html<\/code> would block
\n https:\/\/www.example.com\/private_file.html<\/code>, but would allow
\n https:\/\/www.example.com\/Private_File.html<\/code>.\n<\/p>\nHow do I block Googlebot?<\/h2>\n
\n the User-Agent line of an entry.\n<\/p>\n\n
\n web index<\/a>.\n <\/li>\n
\n mobile index<\/a>.\n <\/li>\n
\n image index<\/a>.\n <\/li>\n
\n AdSense content<\/a>
\n (used only if you show AdSense ads on your site).\n <\/li>\n<\/ul>\nCan I allow pages?<\/h2>\n
Allow<\/code>. This
\n extension may not be recognized by all other search engine bots, so check with other search
\n engines you’re interested in to find out. The Allow<\/code> line works exactly like the
\n Disallow<\/code> line. Simply list a directory or page you want to allow.\n<\/p>\nDisallow<\/code> and Allow<\/code> together. For instance, to block
\n access to all pages in a subdirectory except one, you could use the following entries:\n<\/p>\n
\n myfile.html<\/code>.\n<\/p>\n\n I don’t want certain pages of my site to be indexed, but I want to show AdSense ads on those
\n pages. Can I do that?
\n<\/h2>\nDisallow<\/code> all bots other than Mediapartners-Google<\/code> from
\n those pages. This keeps the pages from being indexed, but lets
\n Googlebot-MediaPartners<\/code> bot analyze the pages to determine the ads to show.
\n Googlebot-MediaPartners<\/code> bot doesn’t share pages with the other Google user-agents.
\n For instance, you could use the following entries:\n<\/p>\nI don’t want to list every file that I want to block. Can I use pattern matching?<\/h2>\n
\n bots may follow it.\n<\/p>\nMatching a sequence of characters<\/h2>\n
*<\/code>) to match a sequence of characters. For instance, to block
\n access to all subdirectories that begin with private<\/code>, you could use the following
\n entry:\n<\/p>\nHow can I make sure that my file blocks and allows what I want it to?<\/h2>\n