{"id":16565,"date":"2026-09-02T11:35:46","date_gmt":"2026-09-02T08:35:46","guid":{"rendered":"http:\/\/localhost:8080\/google-bilgi-bankasi\/robots-txt-intro\/"},"modified":"2026-09-05T02:45:48","modified_gmt":"2026-09-04T23:45:48","slug":"robots-txt-intro","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/","title":{"rendered":"Introduction to robots.txt"},"content":{"rendered":"<div class=\"video-wrapper\" style=\"margin-top: 20px;\">\n<\/div>\n<p>\n  A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.<br \/>\n  This is used mainly to avoid overloading your site with requests; <b>it is not a<br \/>\n  mechanism for keeping a web page out of Google<\/b>. To keep a web page out of Google,<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/block-search-indexing-with-noindex\/\">block indexing with <code>noindex<\/code><\/a><br \/>\n  or password-protect the page.\n<\/p>\n<aside class=\"key-point\">\n<p>\n    <b>If you use a CMS, such as Wix or Blogger<\/b>, you might not need to (or<br \/>\n    be able to) edit your robots.txt file directly. Instead, your CMS might expose a search<br \/>\n    settings page or some other mechanism to tell search engines whether or not to crawl your<br \/>\n    page.\n  <\/p>\n<p>\n    If you want to hide or unhide one of your pages from search engines, search for instructions<br \/>\n    about modifying your page visibility in search engines on your CMS (for example,<br \/>\n    search for &#8220;wix hide page from search engines&#8221;).\n  <\/p>\n<\/aside>\n<h2 id=\"what-is-a-robots.txt-file-used-for\" tabindex=\"-1\">What is a robots.txt file used for?<\/h2>\n<p>\n  A robots.txt file is used primarily to manage crawler traffic to your site, and<br \/>\n  <i>usually<\/i> to keep a file off Google, depending on the file type:\n<\/p>\n<table class=\"responsive\">\n<thead>\n<tr>\n<th colspan=\"2\">robots.txt effect on different file types<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Web page<\/td>\n<td>\n<p>\n          You can use a robots.txt file for web pages (HTML, PDF, or other<br \/>\n          <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/file-types-indexed\/\">non-media formats that Google can read<\/a>),<br \/>\n          to manage crawling traffic if you think your server will be overwhelmed by requests<br \/>\n          from Google&#8217;s crawler, or to avoid crawling unimportant or similar pages on your site.\n        <\/p>\n<aside class=\"warning\">\n<p>\n            <b>Warning<\/b>: Don&#8217;t use a robots.txt file as a means to hide your web pages<br \/>\n            (including PDFs and other text-based formats supported by Google) from Google Search<br \/>\n            results.\n          <\/p>\n<p>\n            If other pages point to your page with descriptive text, Google could still index<br \/>\n            the URL without visiting the page. If you want to block your page from search<br \/>\n            results, use another method such as password protection or<br \/>\n            <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/block-search-indexing-with-noindex\/\"><code>noindex<\/code><\/a>.\n          <\/p>\n<\/aside>\n<p>\n          <b>If your web page is blocked with a robots.txt file<\/b>, its URL can still<br \/>\n          appear in search results, but the search result<br \/>\n          <a href=\"https:\/\/support.google.com\/webmasters\/answer\/7489871\" class=\"external-link\">won&#8217;t have a description<\/a>.<br \/>\n          Image files, video files, PDFs, and other non-HTML files embedded in the blocked page will<br \/>\n          be excluded from crawling, too, unless they&#8217;re referenced by other pages that are allowed<br \/>\n          for crawling. If you see this search result for your page and want to fix it, remove the<br \/>\n          robots.txt entry blocking the page. If you want to hide the page completely from Search,<br \/>\n          use<br \/>\n          <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/removals-overview\/#i-control-the-web-page\">another method<\/a>.\n        <\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>Media file<\/td>\n<td>\n<p>\n          Use a robots.txt file to manage crawl traffic, and also to prevent image, video, and<br \/>\n          audio files from appearing in Google Search results. This won&#8217;t prevent other pages or<br \/>\n          users from linking to your image, video, or audio file.\n        <\/p>\n<ul>\n<li>\n            <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/remove-images-hosted-on-your-site-from-search-results\/\">Read more about preventing images from appearing on Google.<\/a>\n          <\/li>\n<li>\n            <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/video-search\/#remove\">Read more about how to remove or restrict your video files from appearing on Google.<\/a>\n          <\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<tr>\n<td>Resource file<\/td>\n<td>\n        You can use a robots.txt file to block resource files such as unimportant image, script,<br \/>\n        or style files, <strong>if you think that pages loaded without these resources won&#8217;t<br \/>\n        be significantly affected by the loss<\/strong>. However, if the absence of these<br \/>\n        resources make the page harder for Google&#8217;s crawler to understand the page, don&#8217;t block<br \/>\n        them, or else Google won&#8217;t do a good job of analyzing pages that depend on<br \/>\n        those resources.\n      <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2 id=\"understand-the-limitations-of-a-robots.txt-file\" tabindex=\"-1\">Understand the limitations of a robots.txt file<\/h2>\n<p>\n  Before you create or edit a robots.txt file, you should know the limits of this URL blocking<br \/>\n  method. Depending on your goals and situation, you might want to consider other mechanisms to<br \/>\n  ensure your URLs are not findable on the web.\n<\/p>\n<ul>\n<li>\n    <b>robots.txt rules may not be supported by all search engines.<\/b><br \/>\n    The instructions in robots.txt files cannot enforce crawler behavior to your site; it&#8217;s up<br \/>\n    to the crawler to obey them. While Googlebot and other respectable web crawlers obey the<br \/>\n    instructions in a robots.txt file, other crawlers might not. Therefore, if you want to keep<br \/>\n    information secure from web crawlers, it&#8217;s better to use other blocking methods, such as<br \/>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/control-what-you-share-with-google\/\">password-protecting private files on your server<\/a>.\n  <\/li>\n<li>\n    <b>Different crawlers interpret syntax differently.<\/b><br \/>\n    Although respectable web crawlers follow the rules in a robots.txt file, each crawler<br \/>\n    might interpret the rules differently. You should know the<br \/>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-syntax\/#syntax\">proper syntax<\/a> for addressing<br \/>\n    different web crawlers as some might not understand certain instructions.\n  <\/li>\n<li id=\"robotted-but-indexed\">\n    <b>A page that&#8217;s disallowed in robots.txt can<br \/>\n      still be indexed if linked to from other sites.<\/b><br \/>\n    While Google won&#8217;t crawl or index the content blocked by a robots.txt file, we might still<br \/>\n    find and index a disallowed URL if it is linked from other places on the web. As a result,<br \/>\n    the URL address and, potentially, other publicly available information such as anchor text<br \/>\n    in links to the page can still appear in Google Search results. To properly prevent your URL<br \/>\n    from appearing in Google Search results,<br \/>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/control-what-you-share-with-google\/\">password-protect the files on your server<\/a>,<br \/>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/block-search-indexing-with-noindex\/\">use the <code>noindex<\/code> <code>meta<\/code> tag or response header<\/a>,<br \/>\n    or remove the page entirely.\n  <\/li>\n<\/ul>\n<aside class=\"caution\">\n  <b>Caution<\/b>: Combining multiple crawling and indexing rules might cause<br \/>\n  some rules to counteract other rules. Learn how to<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-meta-tag\/#combining\">combine crawling with indexing and serving rules<\/a>.<br \/>\n<\/aside>\n<h2 id=\"create-or-update-a-robots.txt-file\" tabindex=\"-1\">Create or update a robots.txt file<\/h2>\n<p>\n  If you decided that you need one, learn how to<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-to-write-and-submit-a-robots-txt-file\/\">create a robots.txt file<\/a>. Or if<br \/>\n  you already have one, learn how to<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/update-your-robots-txt-file\/\">update it<\/a>.\n<\/p>\n<p>\n    Want to learn more? Check out the following resources:\n  <\/p>\n<ul>\n<li>\n      <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-to-write-and-submit-a-robots-txt-file\/\" class=\"external-link\">How to write and submit a robots.txt file<\/a>\n    <\/li>\n<li>\n      <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/update-your-robots-txt-file\/\" class=\"external-link\">Update your robots.txt file<\/a>\n    <\/li>\n<li>\n      <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-google-interprets-the-robots-txt-specification\/\" class=\"external-link\">How Google interprets the robots.txt specification<\/a>\n    <\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.<\/p>\n","protected":false},"menu_order":28,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[691],"ar_kb_etiket":[532,530,560],"class_list":["post-16565","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-robots-txt-crawling-and-indexing","ar_kb_etiket-crawling","ar_kb_etiket-indexing","ar_kb_etiket-robots-txt"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":2,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16565\/revisions"}],"predecessor-version":[{"id":25571,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16565\/revisions\/25571"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/26329"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=16565"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=16565"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=16565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}