Using a robots.txt file

February 24, 2006

A couple of weeks ago, we
launched a robots.txt
analysis tool. This tool gives you information about how
Googlebot
interprets your robots.txt file. You can read more about the
robots.txt Robots Exclusion Standard,
but we thought we’d answer some common questions here.

What is a robots.txt file?

A robots.txt file provides restrictions to search engine robots (known as “bots”) that crawl the
web. These bots are automated, and before they access pages of a site, they check to see if a
robots.txt file exists that prevents them from accessing certain pages.

Does my site need a robots.txt file?

Only if your site includes content that you don’t want search engines to index. If you want
search engines to index everything in your site, you don’t need a robots.txt file (not even an
empty one).

Where should the robots.txt file be located?

The robots.txt file must reside in the root of the domain. A robots.txt file located in
a subdirectory isn’t valid, as bots only check for this file in the root of the domain. For
instance, https://www.example.com/robots.txt is a valid location. But,
https://www.example.com/mysite/robots.txt is not. If you don’t have access to the
root of a domain, you can restrict access using the
Robots META tag.

How do I create a robots.txt file?

You can create this file in any text editor. It should be an
ASCII-encoded text file,
not an HTML file. The filename should be lowercase.

What should the syntax of my robots.txt file be?

The simplest robots.txt file uses two rules:

• User-Agent: the robot the following rule applies to
• Disallow: the pages you want to block

These two lines are considered a single entry in the file. You can include as many entries as you
want. You can include multiple Disallow lines in one entry.

User-Agent

A user-agent is a specific search engine robot. The
Web Robots Database
lists many common bots. You can set an entry to apply to a specific bot (by listing the name) or
you can set it to apply to all bots (by listing an asterisk). An entry that applies to all bots
looks like this:

Disallow

The Disallow line lists the pages you want to block. You can list a specific URL or
a pattern. The entry should begin with a forward slash (/).

• To block the entire site, use a forward slash: Disallow: /
• To block a directory, follow the directory name with a forward slash:
  Disallow: /private_directory/
• To block a page, list the page:
  Disallow: /private_file.html

URLs are case-sensitive. For instance, Disallow: /private_file.html would block
https://www.example.com/private_file.html, but would allow
https://www.example.com/Private_File.html.

How do I block Googlebot?

Google uses several user-agents. You can block access to any of them by including the bot name on
the User-Agent line of an entry.

• Googlebot: crawl pages from our
  web index.
• Googlebot-Mobile: crawls pages for our
  mobile index.
• Googlebot-Image: crawls pages for our
  image index.
• Mediapartners-Google: crawls pages to determine
  AdSense content
  (used only if you show AdSense ads on your site).

Can I allow pages?

Yes, Googlebot recognizes an extension to the robots.txt standard called Allow. This
extension may not be recognized by all other search engine bots, so check with other search
engines you’re interested in to find out. The Allow line works exactly like the
Disallow line. Simply list a directory or page you want to allow.

You may want to use Disallow and Allow together. For instance, to block
access to all pages in a subdirectory except one, you could use the following entries:

Those entries would block all pages inside the folder1 directory except for
myfile.html.

I don’t want certain pages of my site to be indexed, but I want to show AdSense ads on those
pages. Can I do that?

Yes, you can Disallow all bots other than Mediapartners-Google from
those pages. This keeps the pages from being indexed, but lets
Googlebot-MediaPartners bot analyze the pages to determine the ads to show.
Googlebot-MediaPartners bot doesn’t share pages with the other Google user-agents.
For instance, you could use the following entries:

I don’t want to list every file that I want to block. Can I use pattern matching?

Yes, Googlebot interprets some pattern matching. This is an extension of the standard, so not all
bots may follow it.

Matching a sequence of characters

You can use an asterisk (*) to match a sequence of characters. For instance, to block
access to all subdirectories that begin with private, you could use the following
entry:

How can I make sure that my file blocks and allows what I want it to?

You can use our
robots.txt analysis tool to:

• Check specific URLs to see if your robots.txt file allows or blocks them.
• See if Googlebot had trouble parsing any lines in your robots.txt file.
• Test changes to your robots.txt file.

Also, if you don’t currently use a robots.txt file, you can create one and then test it with the
tool before you upload it to your site.

If I change my robots.txt file or upload a new one, how soon will it take effect?

We generally download robots.txt files about once a day. You can see the last time we downloaded
your file by accessing the robots.txt tab in your Sitemaps account and checking the
Last downloaded date and time.

Posted by Vanessa Fox