{"id":25345,"date":"2025-03-07T00:00:00","date_gmt":"2025-03-07T00:00:00","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-refresher-robots-txt-a-flexible-way-to-control-how-machines-explore-your-website\/"},"modified":"2025-03-07T00:00:00","modified_gmt":"2025-03-07T00:00:00","slug":"robots-refresher-robots-txt-a-flexible-way-to-control-how-machines-explore-your-website","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-refresher-robots-txt-a-flexible-way-to-control-how-machines-explore-your-website\/","title":{"rendered":"Robots Refresher: robots.txt \u2014 a flexible way to control how machines explore your website"},"content":{"rendered":"

Friday, March 7, 2025<\/p>\n

\n A long-standing tool for website owners, robots.txt has been in active use for over 30 years and
\n is broadly supported by crawler operators (such as tools for site owners, services, and search
\n engines). In this edition of the robots refresher series<\/a>,
\n we’ll take a closer look at robots.txt as a flexible way to tell robots what you want them to do
\n (or not do) on your website.\n<\/p>\n

Getting started with robots.txt<\/h2>\n

\n The way these files work is simple: you make a text file called “robots.txt” and then upload it to
\n your website—and if you’re using a content management system (CMS), it’s likely even easier. You
\n can leave your robots.txt file empty (or not have one at all) if your whole
\n site may be crawled, or you can add rules to manage crawling. For example, to tell all bots (also
\n known as crawlers, robots, spiders) to stay out of your “add to cart” page, you could write this
\n in your robots.txt file:\n<\/p>\n

<\/div>\n

More specific things you can do with robots.txt<\/h2>\n

\n robots.txt is the Swiss Army knife of expressing what you want different robots to do or
\n not do on your website: it can be just a few lines, or it can be complex with more elaborate
\n rules targeting very specific URL patterns. You can use a robots.txt file for solving technical
\n issues (such as unnecessary paginated pages<\/a>),
\n or for editorial or personal reasons (such as just not wanting certain things crawled). For
\n example, you could:\n<\/p>\n\n\n\n\n <\/colgroup>\n\n\n\n\n\n
\n

Inform multiple bots (but not all) about the same rule<\/b><\/p>\n

\n This group tells both examplebot<\/code> and otherbot<\/code> to stay away from
\n the \/search<\/code> path.\n <\/p>\n<\/td>\n

<\/span><\/p>\n
<\/div>\n<\/td>\n<\/tr>\n
\n

Tell one bot to avoid paths that contain a specific piece of text<\/b><\/p>\n

\n For example, you could tell documentsbot<\/code> not to crawl any file that contains
\n “.pdf” in its name.\n <\/p>\n<\/td>\n

<\/span><\/p>\n
<\/div>\n<\/td>\n<\/tr>\n
\n

Tell a bot it may crawl your blog, but not the drafts<\/b><\/p>\n<\/td>\n

<\/span><\/p>\n
<\/div>\n<\/td>\n<\/tr>\n
\n

Block a crawler from part of your website, while allowing other crawlers to access your site<\/b><\/p>\n

\n This robots.txt file disallows the mentioned aicorp-trainer-bot<\/code> from accessing
\n anything other than the home page, while allowing other crawlers (such as search engines) to
\n access the site.\n <\/p>\n<\/td>\n

<\/span><\/p>\n
<\/div>\n<\/td>\n<\/tr>\n
\n

Leave a comment for your future self<\/b><\/p>\n

\n You can start a line with #<\/code> to remind yourself about why you put a certain rule there.\n <\/p>\n<\/td>\n

\n <\/span><\/p>\n
<\/div>\n<\/td>\n<\/tr>\n<\/table>\n

\n For even more, you can check out our list of useful robots.txt rules<\/a>.\n<\/p>\n

Making changes to your robots.txt file (practically) <\/h2>\n

\n The Robots Exclusion Protocol (REP)<\/a> works
\n by putting together rules (“allow<\/span>” or “disallow<\/span>“)
\n and specifying which robots these rules apply to. You don’t need to learn programming or fiddle
\n with tools; you can just put these rules in a text file and upload it to your website.\n<\/p>\n

\n For most websites, it’s even simpler than that! If you’re using a CMS, it usually has
\n something already built in to help you change your robots.txt file. For example, some CMSes let
\n you customize your robots.txt file using checkboxes or with a simple form, and many have plugins
\n that help you set up and write rules for your robots.txt file. To check what’s possible within
\n your CMS, you can do a search for the name of your CMS +”edit robots.txt file”.\n<\/p>\n

\n Once you’ve got things set up, you can also test to make sure your file is set up how you
\n intended. There are many testing tools built by the web community to help with this, such as
\n TametheBot’s robots.txt testing tool<\/a>
\n and this robots.txt parser<\/a> that
\n are using the open-source robots.txt parser library<\/a>.\n<\/p>\n

\n If you have any questions about robots.txt, you can find us on LinkedIn<\/a>
\n or chat with like-minded experts in our community forums<\/a>.\n<\/p>\n

Posted by Martin Splitt<\/a>
\n and John Mueller<\/a>,
\n Search relations team<\/p>\n

\n

Check out the rest of the Robots Refresher series:<\/h2>\n","protected":false},"excerpt":{"rendered":"

Friday, March 7, 2025 A long-standing tool for website owners, robots.txt has been in active use for over 30 years and is broadly supported by crawler operators (such as tools for site owners, services, and search engines). In this edition of the robots refresher series, we’ll take a closer look at robots.txt as a flexible […]<\/p>\n","protected":false},"menu_order":79846,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[665],"ar_kb_etiket":[],"class_list":["post-25345","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-blog"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25345","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25345\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27465"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25345"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25345"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}