{"id":25662,"date":"2026-09-06T00:46:32","date_gmt":"2026-09-05T21:46:32","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-google-interprets-the-robots-txt-specification\/"},"modified":"2026-09-06T00:49:53","modified_gmt":"2026-09-05T21:49:53","slug":"how-google-interprets-the-robots-txt-specification","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-google-interprets-the-robots-txt-specification\/","title":{"rendered":"How Google interprets the robots.txt specification"},"content":{"rendered":"

\n Google’s automated
\n crawlers<\/a> support the
\n Robots Exclusion Protocol (REP)<\/a>.
\n This means that before crawling a site, Google’s crawlers download and parse the site’s
\n robots.txt file to extract information about which parts of the site may be crawled. The REP
\n isn’t applicable to Google’s crawlers that are controlled by users (for example, feed
\n subscriptions), or crawlers that are used to increase user safety (for example, malware
\n analysis).\n<\/p>\n

\n This page describes Google’s interpretation of the REP. For the original
\n standard, check
\n RFC 9309<\/a>.\n<\/p>\n

What is a robots.txt file<\/h2>\n

\n If you don’t want crawlers to access sections of your site, you can create a robots.txt file
\n with appropriate rules. A robots.txt file is a text file containing rules about which
\n crawlers may access which parts of a site. For example, the robots.txt file for example.com
\n may look like this:\n<\/p>\n

<\/div>\n

\n If you’re new to robots.txt, start with our
\n intro to robots.txt<\/a>. You can also find
\n tips for creating a robots.txt file<\/a>.\n<\/p>\n

File location and range of validity<\/h2>\n

\n You must place the robots.txt file in the top-level directory of a site, on a supported
\n protocol. The URL for the robots.txt file is (like other URLs) case-sensitive. In case of
\n Google Search, the supported protocols are HTTP, HTTPS, and FTP. On HTTP and HTTPS, crawlers
\n fetch the robots.txt file with an HTTP non-conditional GET<\/code> request; on FTP,
\n crawlers use a standard RETR (RETRIEVE)<\/code> command, using anonymous login.\n<\/p>\n

\n The rules listed in the robots.txt file apply only to the host, protocol, and port number
\n where the robots.txt file is hosted.\n<\/p>\n

Examples of valid robots.txt URLs<\/h2>\n

The following table contains examples of robots.txt URLs and what URL paths they’re valid for.
\n Column one contains the URL of a robots.txt file, and column two contains domains that that robots.txt file would and wouldn’t apply to.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
Robots.txt URL examples<\/th>\n<\/tr>\n<\/thead>\n
https:\/\/example.com\/robots.txt<\/code><\/td>\n\n

\n This is the general case. It’s not valid for other subdomains, protocols, or port
\n numbers. It’s valid for all files in all subdirectories on the same host, protocol,
\n and port number.\n <\/p>\n

Valid for:<\/span><\/p>\n

    \n
  • https:\/\/example.com\/<\/code><\/li>\n
  • https:\/\/example.com\/folder\/file<\/code><\/li>\n<\/ul>\n

    Not valid for:<\/span><\/p>\n

      \n
    • https:\/\/other.example.com\/<\/code><\/li>\n
    • http:\/\/example.com\/<\/code><\/li>\n
    • https:\/\/example.com:8181\/<\/code><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
https:\/\/www.example.com\/robots.txt<\/code><\/td>\n\n

\n A robots.txt on a subdomain is only valid for that subdomain.\n <\/p>\n

\n Valid for:<\/span>
\n https:\/\/www.example.com\/<\/code>\n <\/p>\n

Not valid for:<\/span><\/p>\n

    \n
  • https:\/\/example.com\/<\/code><\/li>\n
  • https:\/\/shop.www.example.com\/<\/code><\/li>\n
  • https:\/\/www.shop.example.com\/<\/code><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
https:\/\/example.com\/folder\/robots.txt<\/code><\/td>\n\n Not a valid robots.txt file. Crawlers don’t check for robots.txt files in subdirectories.\n <\/td>\n<\/tr>\n
https:\/\/www.exämple.com\/robots.txt<\/code><\/td>\n\n

\n IDNs are equivalent to their punycode versions. See also
\n RFC 3492<\/a>.\n <\/p>\n

Valid for:<\/span><\/p>\n

ftp:\/\/example.com\/robots.txt<\/code><\/td>\n\n

\n Valid for:<\/span>
\n ftp:\/\/example.com\/<\/code>\n <\/p>\n

\n Not valid for:<\/span>
\n https:\/\/example.com\/<\/code>\n <\/p>\n<\/td>\n<\/tr>\n

https:\/\/212.96.82.21\/robots.txt<\/code><\/td>\n\n

\n A robots.txt with an IP-address as the hostname is only valid for crawling of that
\n IP address as the hostname. It isn’t automatically valid for all websites hosted on that
\n IP address (though it’s possible that the robots.txt file is shared, in which case it
\n would also be available under the shared hostname).\n <\/p>\n

\n Valid for:<\/span>
\n https:\/\/212.96.82.21\/<\/code>\n <\/p>\n

\n Not valid for:<\/span>
\n https:\/\/example.com\/<\/code> (even if hosted on 212.96.82.21<\/code>)\n <\/p>\n<\/td>\n<\/tr>\n

https:\/\/example.com:443\/robots.txt<\/code><\/td>\n\n

\n Standard port numbers (80<\/code> for HTTP, 443<\/code> for HTTPS,
\n 21<\/code> for FTP) are equivalent to their default hostnames.\n <\/p>\n

Valid for:<\/span><\/p>\n

    \n
  • https:\/\/example.com:443\/<\/code><\/li>\n
  • https:\/\/example.com\/<\/code><\/li>\n<\/ul>\n

    \n Not valid for:<\/span>
    \n https:\/\/example.com:444\/<\/code>\n <\/p>\n<\/td>\n<\/tr>\n

https:\/\/example.com:8181\/robots.txt<\/code><\/td>\n\n

\n Robots.txt files on non-standard port numbers are only valid for content made
\n available through those port numbers.\n <\/p>\n

\n Valid for:<\/span>
\n https:\/\/example.com:8181\/<\/code>\n <\/p>\n

\n Not valid for:<\/span>
\n https:\/\/example.com\/<\/code>\n <\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Handling of errors and HTTP status codes<\/h2>\n

\n When requesting a robots.txt file, the HTTP status code of the server’s response affects how
\n the robots.txt file will be used by Google’s crawlers. The following table summarizes how
\n Googlebot treats robots.txt files for different HTTP status codes.\n<\/p>\n\n\n\n\n\n\n\n\n\n
Handling of errors and HTTP status codes<\/th>\n<\/tr>\n<\/thead>\n
2xx (success)<\/code><\/td>\n\n HTTP status codes that signal success prompt Google’s crawlers to process the robots.txt
\n file as provided by the server.\n <\/td>\n<\/tr>\n
3xx (redirection)<\/code><\/td>\n\n

\n Google follows at least five redirect hops as defined by
\n RFC 1945<\/a> and then
\n stops and treats it as a 404<\/code> for the robots.txt file. This also applies to any
\n disallowed URLs in the redirect chain, since the crawler couldn’t fetch rules due to
\n the redirects.\n <\/p>\n

\n Google doesn’t follow logical redirects in robots.txt files (frames, JavaScript, or
\n meta refresh-type redirects).\n <\/p>\n<\/td>\n<\/tr>\n

4xx (client errors)<\/code><\/td>\n\n

\n Google’s crawlers treat all 4xx<\/code> errors, except 429<\/code>, as if a
\n valid robots.txt file didn’t exist. This means that Google assumes that there are no crawl
\n restrictions.\n <\/p>\n

5xx (server errors)<\/code><\/td>\n\n

\n If Google finds a robots.txt file but can’t fetch it, Google follows this behavior:\n <\/p>\n

    \n
  1. For the first 12 hours, Google stops crawling the site but keeps trying to fetch the
    \n robots.txt file.\n <\/li>\n
  2. If Google can’t fetch a new version, for the next 30 days Google will use the last good
    \n version, while still trying to fetch a new version. A 503 (service unavailable)<\/code>
    \n error results in fairly frequent retrying. If there’s no cached version available,
    \n Google assumes there’s no crawl restrictions.\n <\/li>\n
  3. If the errors are still not fixed after 30 days:\n
      \n
    • If the site is generally available to Google, Google will behave as if there is no
      \n robots.txt file (but still keep checking for a new version).\n <\/li>\n
    • \n If the site has general availability problems, Google will stop crawling the site,
      \n while still periodically requesting a robots.txt file.\n <\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
Other errors<\/td>\n\n A robots.txt file which cannot be fetched due to DNS or networking issues, such as
\n timeouts, invalid responses, reset or interrupted connections, and HTTP chunking errors,
\n is treated as a server error<\/a>.\n <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Caching<\/h2>\n

\n Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it
\n longer in situations where refreshing the cached version isn’t possible (for example, due to
\n timeouts or 5xx<\/code> errors). The cached response may be shared by different crawlers.
\n Google may increase or decrease the cache lifetime based on
\n max-age Cache-Control<\/a>
\n HTTP headers.\n<\/p>\n

File format<\/h2>\n

\n The robots.txt file must be a
\n UTF-8<\/a> encoded plain text
\n file and the lines must be separated by CR<\/code>, CR\/LF<\/code>, or
\n LF<\/code>.\n<\/p>\n

\n Google ignores invalid lines in robots.txt files, including the Unicode
\n Byte Order Mark<\/a>
\n (BOM) at the beginning of the robots.txt file, and use only valid lines. For example, if the
\n content downloaded is HTML instead of robots.txt rules, Google will try to parse the content
\n and extract rules, and ignore everything else.\n<\/p>\n

\n Similarly, if the character encoding of the robots.txt file isn’t UTF-8, Google may ignore
\n characters that are not part of the UTF-8 range, potentially rendering robots.txt rules
\n invalid.\n<\/p>\n

\n Google enforces a robots.txt file size limit of 500
\n kibibytes<\/a> (KiB). Content
\n which is after the maximum file size is ignored. You can reduce the size of the robots.txt
\n file by consolidating rules that would result in an oversized robots.txt file. For
\n example, place excluded material in a separate directory.\n<\/p>\n

Syntax<\/h2>\n

\n A valid robots.txt line consists of a field, a colon, and a value. Field names are
\n case-insensitive (for example, User-agent<\/code> and user-agent<\/code> are treated the
\n same). Spaces are optional, but recommended to improve readability. Space at the beginning and
\n at the end of the line is ignored. To include comments, precede your comment with the
\n #<\/code> character. Keep in mind that everything after the #<\/code> character will
\n be ignored. The general format is
\n <field>:<value><#optional-comment><\/code>.\n<\/p>\n

\n Google supports the following fields (other fields such as crawl-delay<\/code> aren’t supported):\n<\/p>\n