{"id":23827,"date":"2006-09-19T00:00:00","date_gmt":"2006-09-19T00:00:00","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/debugging-blocked-urls\/"},"modified":"2006-09-19T00:00:00","modified_gmt":"2006-09-19T00:00:00","slug":"debugging-blocked-urls","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/debugging-blocked-urls\/","title":{"rendered":"Debugging blocked URLs"},"content":{"rendered":"

Tuesday, September 19, 2006<\/p>\n

\n Vanessa’s been posting a lot lately, and I’m starting to feel left out. So here my tidbit of
\n wisdom for you: I’ve noticed a couple of webmasters confused by
\n “blocked by robots.txt<\/a>”
\n errors, and I wanted to share the steps I take when debugging
\n robots.txt<\/a> problems:\n<\/p>\n

A handy checklist for debugging a blocked URL<\/h2>\n

\n Let’s assume you are looking at crawl errors for your website and notice a URL restricted by
\n robots.txt that you weren’t intending to block:\n<\/p>\n

<\/div>\n

Check the robots.txt analysis tool<\/h2>\n

\n The first thing you should do is go to the
\n robots.txt analysis tool<\/a>
\n for that site. Make sure you are looking at the correct site for that URL, paying attention that
\n you are looking at the right protocol and subdomain. (Subdomains and protocols may have their own
\n robots.txt file, so https:\/\/www.example.com\/robots.txt<\/code> may be different from
\n https:\/\/example.com\/robots.txt<\/code> and may be different from
\n https:\/\/amanda.example.com\/robots.txt.<\/code>) Paste the blocked URL into the “Test URLs
\n against this robots.txt file” box. If the tool reports that it is blocked, you’ve found your
\n problem. If the tool reports that it’s allowed, we need to investigate further.\n<\/p>\n

\n At the top of the robots.txt analysis tool, take a look at the
\n HTTP status code<\/a>.
\n If we are reporting anything other than a 200 (Success)<\/code> or a
\n 404 (Not found)<\/code> then we may not be able to reach your robots.txt file, which stops
\n our crawling process. (Note that you can see the last time we downloaded your robots.txt file at
\n the top of this tool. If you make changes to your file, check this date and time to see if your
\n changes were made after our last download.)\n<\/p>\n

Check for changes in your robots.txt file<\/h2>\n

\n If these look fine, you may want to check and see if your robots.txt file has changed since the
\n error occurred by checking the date to see when your robots.txt file was last modified. If it
\n was modified after the date given for the error in the crawl errors, it might be that someone
\n has changed the file so that the new version no longer blocks this URL.\n<\/p>\n

Check for redirects of the URL<\/h2>\n

\n If you can be certain that this URL isn’t blocked, check to see if the URL redirects to another
\n page. When Googlebot fetches a URL, it checks the robots.txt file to make sure it is allowed to
\n access the URL. If the robots.txt file allows access to the URL, but the URL returns a redirect,
\n Googlebot checks the robots.txt file again to see if the destination URL is accessible. If at any
\n point Googlebot is redirected to a blocked URL, it reports that it could not get the content of
\n the original URL because it was blocked by robots.txt.\n<\/p>\n

\n Sometimes this behavior is easy to spot because a particular URL always redirects to another
\n one. But sometimes this can be tricky to figure out. For instance:\n<\/p>\n