{"id":25672,"date":"2026-09-06T00:47:45","date_gmt":"2026-09-05T21:47:45","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-to-write-and-submit-a-robots-txt-file\/"},"modified":"2026-09-06T00:51:44","modified_gmt":"2026-09-05T21:51:44","slug":"how-to-write-and-submit-a-robots-txt-file","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-to-write-and-submit-a-robots-txt-file\/","title":{"rendered":"How to write and submit a robots.txt file"},"content":{"rendered":"<aside class=\"key-point\">\n<p>\n    <b>If you use a site hosting service, such as Wix or Blogger<\/b>, you might not need to (or<br \/>\n    be able to) edit your robots.txt file directly. Instead, your provider might expose a search<br \/>\n    settings page or some other mechanism to tell search engines whether or not to crawl your<br \/>\n    page.\n  <\/p>\n<p>\n    If you want to hide or unhide one of your pages from search engines, search for instructions<br \/>\n    about modifying your page visibility in search engines on your hosting service, for example,<br \/>\n    search for &#8220;wix hide page from search engines&#8221;.\n  <\/p>\n<\/aside>\n<p>\n  You can<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/\">control which files crawlers may access<\/a><br \/>\n  on your site with a robots.txt file.\n<\/p>\n<p>\n  A robots.txt file lives at the root of your site. So, for site <code>www.example.com<\/code>,<br \/>\n  the robots.txt file lives at <code>www.example.com\/robots.txt<\/code>. robots.txt is a plain<br \/>\n  text file that follows the<br \/>\n  <a href=\"https:\/\/en.wikipedia.org\/wiki\/Robots_exclusion_standard#About_the_standard\" class=\"external-link\">Robots Exclusion Standard<\/a>.<br \/>\n  A robots.txt file consists of one or more rules. Each rule blocks or allows access for all or<br \/>\n  a specific crawler to a specified file path on the domain or subdomain where the robots.txt<br \/>\n  file is hosted. Unless you specify otherwise in your robots.txt file, all files are implicitly<br \/>\n  allowed for crawling.\n<\/p>\n<p>Here is a simple robots.txt file with two rules:<\/p>\n<div><\/div>\n<p><b>Here&#8217;s what that robots.txt file means:<\/b><\/p>\n<ol>\n<li>\n    The user agent named Googlebot is not allowed to crawl any URL that starts with<br \/>\n    <code>https:\/\/example.com\/nogooglebot\/<\/code>.\n  <\/li>\n<li>\n    All other user agents are allowed to crawl the entire site. This could have been omitted<br \/>\n    and the result would be the same; the default behavior is that user agents are allowed to<br \/>\n    crawl the entire site.\n  <\/li>\n<li>\n    The site&#8217;s <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/sitemaps-overview\/\">sitemap file<\/a> is located at<br \/>\n    <code>https:\/\/www.example.com\/sitemap.xml<\/code>.\n  <\/li>\n<\/ol>\n<p>See the <a href=\"#create_rules\">syntax<\/a> section for more examples.<\/p>\n<h2 id=\"learn_robots_txt_syntax\" tabindex=\"-1\">Basic guidelines for creating a robots.txt file<\/h2>\n<p>\n  Creating a robots.txt file and making it generally accessible and useful involves four steps:\n<\/p>\n<ol>\n<li><a href=\"#format_location\">Create a file named robots.txt<\/a>.<\/li>\n<li><a href=\"#create_rules\">Add rules to the robots.txt file<\/a>.<\/li>\n<li><a href=\"#upload\">Upload the robots.txt file to the root of your site<\/a>.<\/li>\n<li><a href=\"#testing\">Test the robots.txt file<\/a>.<\/li>\n<\/ol>\n<h2 id=\"format_location\" tabindex=\"-1\">Create a robots.txt file<\/h2>\n<p>\n  You can use almost any text editor to create a robots.txt file. For example, Notepad,<br \/>\n  TextEdit, vi, and emacs can create valid robots.txt files. Don&#8217;t use a word processor; word<br \/>\n  processors often save files in a proprietary format and can add unexpected characters, such as<br \/>\n  curly quotes, which can cause problems for crawlers. Make sure to save the file with UTF-8<br \/>\n  encoding if prompted during the save file dialog.\n<\/p>\n<p><b>Format and location rules:<\/b><\/p>\n<ul>\n<li>The file must be named robots.txt.<\/li>\n<li>Your site can have only one robots.txt file.<\/li>\n<li>\n    The robots.txt file must be located at the root of the site host to<br \/>\n    which it applies. For instance, to control crawling on all URLs below<br \/>\n    <code>https:\/\/www.example.com\/<\/code>, the robots.txt file must be located at<br \/>\n    <code>https:\/\/www.example.com\/robots.txt<\/code>. It <i>cannot<\/i> be placed in a<br \/>\n    subdirectory (for example, at <code>https:\/\/example.com\/pages\/robots.txt<\/code>). If you&#8217;re<br \/>\n    unsure about how to access your site root, or need permissions to do so, contact your web<br \/>\n    hosting service provider. If you can&#8217;t access your site root, use an alternative blocking<br \/>\n    method such as <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/block-search-indexing-with-noindex\/\"><code>meta<\/code> tags<\/a>.\n  <\/li>\n<li>\n    A robots.txt file can be posted on a subdomain (for example,<br \/>\n    <code>https:\/\/<b>site<\/b>.example.com\/robots.txt<\/code>) or on non-standard<br \/>\n    ports (for example, <code>https:\/\/example.com:<b>8181<\/b>\/robots.txt<\/code>).\n  <\/li>\n<li>A robots.txt file applies only to paths within the protocol, host, and port where it is<br \/>\n    posted. That is, rules in <code>https:\/\/example.com\/robots.txt<\/code> apply only to files in<br \/>\n    <code>https:\/\/example.com\/<\/code>, not to subdomains such as<br \/>\n    <code>https:\/\/m.example.com\/<\/code>, or alternate protocols, such as<br \/>\n    <code>http:\/\/example.com\/<\/code>.\n  <\/li>\n<li>\n    A robots.txt file must be a UTF-8 encoded text file (which includes ASCII). Google may<br \/>\n    ignore characters that are not part of the UTF-8 range, potentially rendering robots.txt<br \/>\n    rules invalid.\n  <\/li>\n<\/ul>\n<h2 id=\"create_rules\" tabindex=\"-1\">How to write robots.txt rules<\/h2>\n<p>\n  Rules are instructions for crawlers about which parts of your site they can crawl. Follow<br \/>\n  these guidelines when adding rules to your robots.txt file:\n<\/p>\n<ul>\n<li>A robots.txt file consists of one or more groups (set of rules).<\/li>\n<li>\n    Each group consists of multiple rules (also known as directives), one rule per line. Each<br \/>\n    group begins with a <code>User-agent<\/code> line that specifies the target of the groups.\n  <\/li>\n<li>A group gives the following information:\n<ul>\n<li>Who the group applies to (the user agent).<\/li>\n<li>Which directories or files that agent <i>can<\/i> access.<\/li>\n<li>Which directories or files that agent <i>cannot<\/i> access.<\/li>\n<\/ul>\n<\/li>\n<li>\n    Crawlers process groups from top to bottom. A user agent can match only one rule set, which<br \/>\n    is the first, most specific group that matches a given user agent. If there are multiple<br \/>\n    groups for the same user agent, the groups will be combined into a single group before<br \/>\n    processing.\n  <\/li>\n<li>\n    The default assumption is that a user agent can crawl any page or directory not blocked by a<br \/>\n    <code>disallow<\/code> rule.\n  <\/li>\n<li>\n    Rules are case-sensitive. For instance, <code>disallow: \/file.asp<\/code> applies to<br \/>\n    <code>https:\/\/www.example.com\/file.asp<\/code>, but not<br \/>\n    <code>https:\/\/www.example.com\/FILE.asp<\/code>.\n  <\/li>\n<li>\n    The <code>#<\/code> character marks the beginning of a comment. Comments are ignored during<br \/>\n    processing.\n  <\/li>\n<\/ul>\n<p><b>Google&#8217;s crawlers support the following rules in robots.txt files:<\/b><\/p>\n<ul>\n<li>\n    <code>user-agent:<\/code> [Required, one or more per group] The<br \/>\n    rule specifies the name of the automatic client known as search engine crawler that<br \/>\n    the rule applies to. This is the first line for any rule group. Google user agent names are<br \/>\n    listed in the<br \/>\n    <a href=https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/overview-google-crawlers>Google list of user agents<\/a>.<br \/>\n    Using an asterisk (<code>*<\/code>) matches all crawlers except the various AdsBot crawlers,<br \/>\n    which must be named explicitly. For example:<\/p>\n<div><\/div>\n<\/li>\n<li>\n    <code>disallow:<\/code> [At least one or more <code>disallow<\/code> veya<br \/>\n    <code>allow<\/code> entries per rule] A directory or page, relative to the root domain,<br \/>\n    that you don&#8217;t want the user agent to crawl. If the rule refers to a page, it must be the<br \/>\n    full page name as shown in the browser. It must start with a <code>\/<\/code> character and if<br \/>\n    it refers to a directory, it must end with the <code>\/<\/code> mark.\n  <\/li>\n<li>\n    <code>allow:<\/code> [At least one or more <code>disallow<\/code> veya<br \/>\n    <code>allow<\/code> entries per rule] A directory or page, relative to the root domain,<br \/>\n    that may be crawled by the user agent just mentioned. This is used to override a<br \/>\n    <code>disallow<\/code> rule to allow crawling of a subdirectory or page in a disallowed<br \/>\n    directory. For a single page, specify the full page name as shown in the browser. It must<br \/>\n    start with a <code>\/<\/code> character and if it refers to a directory, it must end with the<br \/>\n    <code>\/<\/code> mark.\n  <\/li>\n<li>\n    <code>sitemap:<\/code> [Optional, zero or more per file] The<br \/>\n    location of a sitemap for this site. The sitemap URL must be a fully-qualified URL;<br \/>\n    Google doesn&#8217;t assume or check http\/https\/www.non-www alternates. Sitemaps are a good way to<br \/>\n    indicate which content Google should crawl, as opposed to which content it<br \/>\n    <i>can<\/i> veya <i>cannot<\/i> crawl.<br \/>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/sitemaps-overview\/\">Learn more about sitemaps.<\/a><br \/>\n    <b>Example:<\/b><\/p>\n<div><\/div>\n<\/li>\n<\/ul>\n<p>\n  All rules, except <code>sitemap<\/code>, support the <code>*<\/code> wildcard for a path<br \/>\n  prefix, suffix, or entire string.\n<\/p>\n<p>Lines that don&#8217;t match any of these rules are ignored.<\/p>\n<p>\n  Read our page about<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-google-interprets-the-robots-txt-specification\/\">Google&#8217;s interpretation of the robots.txt specification<\/a><br \/>\n  for the complete description of each rule.\n<\/p>\n<h2 id=\"upload\" tabindex=\"-1\">Upload the robots.txt file<\/h2>\n<p>\n  Once you saved your robots.txt file to your computer, you&#8217;re ready to make it available to<br \/>\n  search engine crawlers. There&#8217;s no one tool that can help you with this, because how you<br \/>\n  upload the robots.txt file to your site depends on your site and server architecture. Get in<br \/>\n  touch with your hosting company or search the documentation of your hosting company; for<br \/>\n  example, search for &#8220;upload files infomaniak&#8221;.\n<\/p>\n<p>\n  After you upload the robots.txt file, test whether it&#8217;s publicly accessible and if Google can<br \/>\n  parse it.\n<\/p>\n<h2 id=\"testing\" tabindex=\"-1\">\n  Test robots.txt markup<br \/>\n<\/h2>\n<p>\n  To test whether your newly uploaded robots.txt file is publicly accessible, open a<br \/>\n  <a href=\"https:\/\/support.google.com\/chrome\/answer\/95464\" class=\"external-link\">private browsing window<\/a><br \/>\n  (or equivalent) in your browser and navigate to the location of the robots.txt file. For<br \/>\n  example, <code>https:\/\/example.com\/robots.txt<\/code>. If you see the contents of your<br \/>\n  robots.txt file, you&#8217;re ready to test the markup.\n<\/p>\n<p>Google offers two options for fixing issues with robots.txt markup:<\/p>\n<ol>\n<li>\n    The <a href=\"https:\/\/support.google.com\/webmasters\/answer\/6062598\" class=\"external-link\"><br \/>\n    robots.txt report<\/a> in Search Console. You can only use this report for robots.txt files<br \/>\n    that are already accessible on your site.\n  <\/li>\n<li>\n    If you&#8217;re a developer, check out and build<br \/>\n    <a href=\"https:\/\/github.com\/google\/robotstxt\" class=\"external-link\">Google&#8217;s open source robots.txt library<\/a>,<br \/>\n    which is also used in Google Search. You can use this tool to test robots.txt files locally<br \/>\n    on your computer.\n  <\/li>\n<\/ol>\n<h2 id=\"submit\" tabindex=\"-1\">Submit robots.txt file to Google<\/h2>\n<p>\n  Once you uploaded and tested your robots.txt file, Google&#8217;s crawlers will automatically find<br \/>\n  and start using your robots.txt file. You don&#8217;t have to do anything. If you updated your<br \/>\n  robots.txt file and you need to refresh Google&#8217;s cached copy as soon as possible, learn<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/update-your-robots-txt-file\/\">how to submit an updated robots.txt file<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A robots.txt file lives at the root of your site. Learn how to create a robots.txt file, see examples, and explore robots.txt rules.<\/p>\n","protected":false},"menu_order":7000,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[708],"ar_kb_etiket":[],"class_list":["post-25672","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-crawling-infrastructure-crawling-and-indexing"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25672","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25672\/revisions"}],"predecessor-version":[{"id":25693,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25672\/revisions\/25693"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27667"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25672"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25672"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25672"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}