How Google interprets the robots.txt specification

Google’s automated
crawlers support the
Robots Exclusion Protocol (REP).
This means that before crawling a site, Google’s crawlers download and parse the site’s
robots.txt file to extract information about which parts of the site may be crawled. The REP
isn’t applicable to Google’s crawlers that are controlled by users (for example, feed
subscriptions), or crawlers that are used to increase user safety (for example, malware
analysis).

This page describes Google’s interpretation of the REP. For the original
standard, check
RFC 9309.

What is a robots.txt file

If you don’t want crawlers to access sections of your site, you can create a robots.txt file
with appropriate rules. A robots.txt file is a text file containing rules about which
crawlers may access which parts of a site. For example, the robots.txt file for example.com
may look like this:

If you’re new to robots.txt, start with our
intro to robots.txt. You can also find
tips for creating a robots.txt file.

File location and range of validity

You must place the robots.txt file in the top-level directory of a site, on a supported
protocol. The URL for the robots.txt file is (like other URLs) case-sensitive. In case of
Google Search, the supported protocols are HTTP, HTTPS, and FTP. On HTTP and HTTPS, crawlers
fetch the robots.txt file with an HTTP non-conditional GET request; on FTP,
crawlers use a standard RETR (RETRIEVE) command, using anonymous login.

The rules listed in the robots.txt file apply only to the host, protocol, and port number
where the robots.txt file is hosted.

Examples of valid robots.txt URLs

The following table contains examples of robots.txt URLs and what URL paths they’re valid for.
Column one contains the URL of a robots.txt file, and column two contains domains that that robots.txt file would and wouldn’t apply to.

Handling of errors and HTTP status codes

When requesting a robots.txt file, the HTTP status code of the server’s response affects how
the robots.txt file will be used by Google’s crawlers. The following table summarizes how
Googlebot treats robots.txt files for different HTTP status codes.

Caching

Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it
longer in situations where refreshing the cached version isn’t possible (for example, due to
timeouts or 5xx errors). The cached response may be shared by different crawlers.
Google may increase or decrease the cache lifetime based on
max-age Cache-Control
HTTP headers.

File format

The robots.txt file must be a
UTF-8 encoded plain text
file and the lines must be separated by CR, CR/LF, or
LF.

Google ignores invalid lines in robots.txt files, including the Unicode
Byte Order Mark
(BOM) at the beginning of the robots.txt file, and use only valid lines. For example, if the
content downloaded is HTML instead of robots.txt rules, Google will try to parse the content
and extract rules, and ignore everything else.

Similarly, if the character encoding of the robots.txt file isn’t UTF-8, Google may ignore
characters that are not part of the UTF-8 range, potentially rendering robots.txt rules
invalid.

Google enforces a robots.txt file size limit of 500
kibibytes (KiB). Content
which is after the maximum file size is ignored. You can reduce the size of the robots.txt
file by consolidating rules that would result in an oversized robots.txt file. For
example, place excluded material in a separate directory.

Syntax

A valid robots.txt line consists of a field, a colon, and a value. Field names are
case-insensitive (for example, User-agent and user-agent are treated the
same). Spaces are optional, but recommended to improve readability. Space at the beginning and
at the end of the line is ignored. To include comments, precede your comment with the
# character. Keep in mind that everything after the # character will
be ignored. The general format is
<field>:<value><#optional-comment>.

Google supports the following fields (other fields such as crawl-delay aren’t supported):

• user-agent: identifies which crawler the rules apply to.
• allow: a URL path that may be crawled.
• disallow: a URL path that may not be crawled.
• sitemap: the complete URL of a sitemap.

The allow and disallow fields are also called rules (also known as
directives). These rules are always specified in the form of
rule: [path] where [path] is optional. By default, there are no
restrictions for crawling for the designated crawlers. Crawlers ignore rules without a
[path].

The [path] value, if specified, is relative to the root of the website from where
the robots.txt file was fetched (using the same protocol, port number, host and domain names).
The path value must start with / to designate the root and the value is
case-sensitive. Learn more about
URL matching based on path values.

user-agent

The user-agent line identifies which crawler rules apply to. See
Google’s crawlers and user-agent strings
for a comprehensive list of user-agent strings you can use in your robots.txt file.

Both the user-agent field name and its value are case-insensitive.

disallow

The disallow rule specifies paths that must not be accessed by the crawlers
identified by the user-agent line the disallow rule is grouped with.
Crawlers ignore the rule without a path.

Google can’t index the content of pages which are disallowed for crawling, but it may still
index the URL and show it in search results without a snippet. Learn how to
block indexing.

The field name (disallow) is case-insensitive, but its value is case-sensitive.

Usage:

allow

The allow rule specifies paths that may be accessed by the designated
crawlers. When no path is specified, the rule is ignored.

The field name (allow) is case-insensitive, but its value is case-sensitive.

Usage:

sitemap

Google, Bing, and other major search engines support the sitemap field in
robots.txt, as defined by sitemaps.org.

The field name (sitemap) is case-insensitive, but its value is case-sensitive.

Usage:

The [absoluteURL] line points to the location of a sitemap or sitemap index file.
It must be a fully qualified URL, including the protocol and host, and doesn’t have to be
URL-encoded. The URL doesn’t have to be on the same host as the robots.txt file. You can
specify multiple sitemap fields, with no limit to the number of sitemaps you can
include. The sitemap field isn’t tied to any specific user agent and may be followed by all
crawlers, provided it isn’t disallowed for crawling.

For example:

Grouping of lines and rules

You can group together rules that apply to multiple user agents by repeating
user-agent lines for each crawler.

For example:

In this example there are four distinct rule groups:

• One group for user agent “a”.
• One group for user agent “b”.
• One group for both “e” and “f” user agents.
• One group for user agent “h”.

For the technical description of a group, see
section 2.1 of the REP.

Order of precedence for user agents

Only one group is valid for a particular crawler. Google’s crawlers determine the correct
group of rules by finding in the robots.txt file the group with the most specific user agent
that matches the crawler’s user agent. Other groups are ignored. All non-matching text is
ignored (for example, both googlebot/1.2 and googlebot* are
equivalent to googlebot). The order of the groups within the robots.txt file is
irrelevant.

If there’s more than one specific group declared for a user agent, all the rules from the
groups applicable to the specific user agent are combined internally into a single group. User
agent specific groups and global groups (*) are not combined.

Examples

Matching of user-agent fields

This is how the crawlers would choose the relevant group:

Grouping of rules

If there are multiple groups in a robots.txt file that are relevant to a specific user agent,
Google’s crawlers internally merge the groups. For example:

The crawlers internally group the rules based on user agent, for example:

Rules other than allow, disallow, and user-agent are
ignored by the robots.txt parser. This means that the following robots.txt snippet is treated
as one group, and thus both user-agent a and b are
affected by the disallow: / rule:

When the crawlers process the robots.txt rules, they ignore the sitemap line.
For example, this is how the crawlers would understand the previous robots.txt snippet:

URL matching based on path values

Google uses the path value in the allow and disallow rules as a
basis to determine whether or not a rule applies to a specific URL on a site. This works by
comparing the rule to the path component of the URL that the crawler is trying to fetch.
The path in the rules can be either raw UTF-8 characters or percent-encoded UTF-8 strings
(per RFC 3986).
As defined in RFC 9309 Section 2.2.2,
Google’s crawlers compare rules against URLs using their percent-encoded forms, and automatically
canonicalize raw UTF-8 rule paths into percent-encoded forms. This means that, for example,
Disallow: /foo/bar/ツ and Disallow: /foo/bar/%E3%83%84 are treated
identically by the parser.

Google, Bing, and other major search engines support a limited form of wildcards for
path values. These wildcard characters are:

• * designates 0 or more instances of any valid character.
• $ designates the end of the URL.

The following table shows how the different wildcard characters affect parsing:

Order of precedence for rules

When matching robots.txt rules to URLs, crawlers use the most specific rule based on the
length of the rule path. In case of conflicting rules, including those with wildcards, Google uses
the least restrictive rule.

The following examples demonstrate which rule Google’s crawlers will apply on a given URL.