A robots.txt file is a standard text document located in the root directory of a website that provides instructions to web crawlers about which pages they should or should not visit. It serves as the primary communication tool between a webmaster and search engine bots, helping manage how resources are used during the discovery process. While it is a fundamental part of technical SEO, many site owners misunderstand its limitations, particularly regarding page indexing and security.
What is the purpose of a robots.txt file?
The primary goal of a robots.txt file is to manage crawl budget. Search engines like Google and Bing do not have infinite resources to crawl every single page on the internet every day. They allocate a specific amount of time and energy to each website, known as the crawl budget. If your site has thousands of low-value pages, such as internal search results, filter parameters, or duplicate content, bots might waste their time on these rather than finding your most important articles or products.
By using a robots.txt file, you can direct bots away from administrative areas, temporary folders, or sensitive search queries. This ensures that the crawler spends its time on high-quality content that you actually want to appear in search results. It is important to remember that these files are public; anyone can see your rules by adding /robots.txt to your domain. Therefore, you should never use this file to hide private information or sensitive directories from users.
How do you structure a robots.txt file correctly?
A standard robots.txt file uses a simple syntax consisting of directives and values. Each set of instructions begins by identifying the bot it applies to, followed by the specific rules for that bot. The file must be named exactly robots.txt, in all lowercase, and placed in the top-level directory of your web server. If it is placed in a subfolder, search engines will not look for it and will assume no rules exist.
The basic components of the file include the following directives:
- User-agent: Identifies which crawler the rule applies to (e.g., Googlebot, Bingbot, or * for all).
- Disallow: Specifies a directory or page that the crawler should not visit.
- Allow: Provides an exception to a Disallow rule, allowing a specific subfolder to be crawled.
- Sitemap: Points the crawler to the location of your XML sitemap to help with discovery.
In a typical account, a simple robots.txt file might look like this: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap_index.xml This tells all bots to stay out of the WordPress admin area but allows them to access the AJAX file, which is often necessary for site functionality.
Why does Google still index pages blocked by robots.txt?
One of the most common points of confusion in SEO is the difference between crawling and indexing. Robots.txt controls crawling, not indexing. If Google finds a link to a page on your site from an external source, it knows that the page exists. Even if your robots.txt file forbids Google from visiting that page, the search engine might still index the URL based on the information it gathers from the external link.
This leads to a specific problem: if you want to keep a page out of search results entirely, you must use a ‘noindex’ meta tag. However, a ‘noindex’ tag only works if the search engine can crawl the page to see the tag. If you block the page in robots.txt, the bot never visits the page, never sees the ‘noindex’ instruction, and the page may remain in the search results. For a page to be removed via ‘noindex’, it must be accessible to the crawler.
Common robots.txt mistakes that hurt search rankings
A single character error in your robots.txt file can have severe consequences. During website migrations or development phases, it is common for developers to use ‘Disallow: /’ to prevent a staging site from being indexed. This directive blocks all crawlers from every page on the site. If this code is accidentally moved to the live production server, your entire website will eventually disappear from search results as bots are told to stop visiting.
Another frequent mistake is blocking internal assets like CSS and JavaScript files. Historically, search engines only cared about text content. Today, Googlebot renders pages like a modern browser to evaluate layout, mobile-friendliness, and user experience. If your robots.txt blocks the folders containing your styles or scripts, the search engine sees a broken version of your site, which can lead to lower rankings and ‘mobile usability’ errors in search consoles.
Case sensitivity is another area where errors occur. While the directives like ‘User-agent’ are not case-sensitive, the paths to files and folders often are. For example, ‘Disallow: /Private/’ will not block a folder named ‘/private/’. It is a best practice to use lowercase for all file and folder names on your server to avoid these technical conflicts.
Managing robots.txt for e-commerce and large websites
Large websites, such as a Shopify store selling thousands of apparel items, face unique challenges. E-commerce platforms often generate a massive number of URLs through faceted navigation, such as filters for color, size, or price range. These filtered pages often contain duplicate content that does not need to be indexed. Without a proper robots.txt strategy, these sites can easily exhaust their crawl budget on useless URL parameters.
For instance, you might use a rule like ‘Disallow: /*?*’ to block all URLs containing a question mark, which often signifies a tracking parameter or a filter. However, you must be careful not to block essential pages that also use parameters. At Alien Road, we suggest reviewing your internal search query logs to identify which URL patterns are being crawled most frequently and determining if they provide actual value to search users.
How to test and validate your robots.txt file?
Before making changes to your live site, you should always validate your rules. Google provides a Robots.txt Tester tool within the older version of Search Console that allows you to paste your code and test specific URLs against it. This helps you see exactly which rules are blocking which pages. There are also various third-party browser extensions and online validators that can highlight syntax errors or logic conflicts.
A common pattern for validation includes these steps:
- Check that the file is accessible at yourdomain.com/robots.txt.
- Ensure there are no ‘Disallow: /’ rules unless you intend to block the entire site.
- Confirm that your CSS and JS folders are not blocked.
- Verify that your XML sitemap URL is correctly listed at the end of the file.
- Test your most important product or landing pages to ensure they are allowed.
Where to start this week
Your first step should be to visit your own website’s /robots.txt address to see what instructions are currently live. Check your Google Search Console ‘Indexing’ report for any warnings that say ‘Indexed, though blocked by robots.txt’. This indicates that you are blocking pages that Google thinks are important enough to index anyway, which is a sign of a logic error in your strategy. If you find these, evaluate whether you should remove the block or add a ‘noindex’ tag while allowing the crawl. Finally, ensure your sitemap is properly linked to help crawlers navigate your site efficiently.
Keep reading: SEO (Search Engine Optimization) · SEO