{"id":25582,"date":"2026-09-05T02:45:55","date_gmt":"2026-09-04T23:45:55","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/troubleshoot-google-search-crawling-errors\/"},"modified":"2026-09-05T02:45:55","modified_gmt":"2026-09-04T23:45:55","slug":"troubleshoot-google-search-crawling-errors","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/troubleshoot-google-search-crawling-errors\/","title":{"rendered":"Troubleshoot Google Search crawling errors"},"content":{"rendered":"<p>Here are the key steps to troubleshooting and fixing Google Search crawling issues for your site:<\/p>\n<ol>\n<li><a href=\"#availability_issues\">See if Googlebot is encountering availability issues on your<br \/>\n    site<\/a>.<\/li>\n<li><a href=\"#not_crawled_should_be\">See whether you have pages that aren&#8217;t being crawled, but<br \/>\n    should be<\/a>.<\/li>\n<li><a href=\"#updates\">See whether any parts of your site need to be crawled more quickly than<br \/>\n    they already are.<\/a><\/li>\n<li><a href=\"#improve_crawl_efficiency\">Improve your site&#8217;s crawl efficiency.<\/a><\/li>\n<li><a href=\"#emergencies\">Handle overcrawling of your site<\/a>.<\/li>\n<\/ol>\n<h2 id=\"availability_issues\" tabindex=\"-1\">See if Googlebot is encountering availability issues on your site<\/h2>\n<p>Improving your site availability won&#8217;t necessarily increase your crawl budget; Google<br \/>\n  determines the best crawl rate based on the crawl demand, as described previously. However,<br \/>\n  availability issues do prevent Google from crawling your site as much as it might want to.<\/p>\n<p><b>Diagnosing:<\/b><\/p>\n<p>\n  Use the <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9679690\" class=\"external-link\">Crawl Stats report<\/a><br \/>\n  to see Googlebot&#8217;s crawling history for your site. The report shows when Google encountered<br \/>\n  availability issues on your site. If availability errors or warnings are reported for your site,<br \/>\n  look for instances in the <b>Host availability<\/b> graphs where Googlebot requests exceeded the<br \/>\n  red limit line, click into the graph to see which URLs were failing, and try to correlate<br \/>\n  those with issues on your site.\n<\/p>\n<p>\n  Additionally, you can also use the<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9012289\" class=\"external-link\">URL Inspection Tool<\/a><br \/>\n  to test a few URLs on your site. If the tool returns<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9012289#live_indexable&#038;zippy=%2Cadditional-response-data%2Curl-status-live-test%2Csite-wide-availability-issues%2Cavailability-live-test\" class=\"external-link\"><b>Hostload exceeded<\/b><\/a><br \/>\n  warnings, that means that Googlebot can&#8217;t crawl as many URLs from your site as it discovered.\n<\/p>\n<p><b>Treating:<\/b><\/p>\n<ul>\n<li>Read the <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9679690\" class=\"external-link\">documentation<br \/>\n  for the Crawl Stats report<\/a> to learn how to find and handle some availability issues.<\/li>\n<li><b>Block pages from crawling if you don&#8217;t want them to be crawled.<\/b> (See <i><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/optimize-your-crawl-budget\/#manage_inventory\" class=\"external-link\">manage<br \/>\n  your inventory<\/a><\/i>)<\/li>\n<li><b>Increase page loading and rendering speed.<\/b> (See <i><a href=\"#improve_crawl_efficiency\">Improve<br \/>\n  your site&#8217;s crawl efficiency<\/a><\/i>)<\/li>\n<li><b>Increase your server capacity.<\/b> If Google consistently seems to be crawling<br \/>\n  your site at its serving capacity limit, but you still have important URLs that aren&#8217;t being<br \/>\n  crawled or updated as much as they need, having more serving resources might enable Google to<br \/>\n  request more pages on your site. Check your host availability history in the<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9679690\" class=\"external-link\">Crawl Stats<br \/>\n    report<\/a> to see if Google&#8217;s crawl rate seems to be crossing the limit line often. If so,<br \/>\n  increase your serving resources for a month and see whether crawling requests increased during<br \/>\n  that same period.<\/li>\n<\/ul>\n<h2 id=\"not_crawled_should_be\" tabindex=\"-1\">See if any parts of your site are not crawled, but should be<\/h2>\n<p>Google spends as much time as necessary on your site in order to index all the high-quality,<br \/>\n  user-valuable content that it can find. If you think that Googlebot is missing important<br \/>\n  content, either it doesn&#8217;t know about the content, the content is blocked from Google, or your<br \/>\n  site availability is throttling Google&#8217;s access (or Google is trying not to overload your site).<\/p>\n<aside class=\"key-point\">Remember the difference between <i>crawling<\/i> and <i>indexing<\/i>.<br \/>\n  This page is about helping Google <i>crawl<\/i> your site efficiently, not whether the pages<br \/>\n  found make it into the index.<\/aside>\n<p><b>Diagnosing:<\/b><\/p>\n<p>Search Console doesn&#8217;t provide a crawl history for your site that can be filtered by URL or<br \/>\n  path, but you can inspect your site logs to see whether specific URLs have been crawled by<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/google-crawlers\/\">Googlebot<\/a>. Whether or<br \/>\n  not those crawled URLs have been indexed is another story.<\/p>\n<p>Remember that for most sites, new pages will take several days minimum to be noticed; most<br \/>\n  sites shouldn&#8217;t expect same-day crawling for URLs, with the exception of time-sensitive sites<br \/>\n  such as news sites.<\/p>\n<p><b>Treating:<\/b><\/p>\n<p>If you are adding pages to your site and they are not being crawled in a reasonable amount of<br \/>\n  time, either Google doesn&#8217;t know about them, the content is blocked, your site has reached its<br \/>\n  maximum serving capacity, or you are <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/optimize-your-crawl-budget\/#more-crawl-budget\" class=\"external-link\">out of crawl budget<\/a>.<\/p>\n<ol>\n<li>Tell Google about your new pages: update your sitemaps to reflect new URLs.<\/li>\n<li>Examine your robots.txt rules to confirm that you&#8217;re not accidentally blocking pages.<\/li>\n<li>Review your crawling priorities (a.k.a. use your crawl budget wisely). <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/optimize-your-crawl-budget\/#manage_inventory\" class=\"external-link\">Manage<br \/>\n  your inventory<\/a> and <a href=\"#improve_crawl_efficiency\">improve your site&#8217;s crawling efficiency<\/a>.<\/li>\n<li><a href=\"#availability_issues\">Check that you&#8217;re not running out of serving capacity<\/a>.<br \/>\n  Googlebot will scale back its crawling if it detects that your servers are having trouble<br \/>\n  responding to crawl requests.<\/li>\n<\/ol>\n<p>Note that pages might not be shown in search results, even if crawled, if there isn&#8217;t<br \/>\n  sufficient value or user demand for the content.<\/p>\n<h2 id=\"updates\" tabindex=\"-1\">See if updates are crawled quickly enough<\/h2>\n<p>If we&#8217;re missing new or updated pages on your site, perhaps it&#8217;s because we haven&#8217;t seen them,<br \/>\n  or haven&#8217;t noticed that they are updated. Here is how you can help us be aware of page<br \/>\n  updates.<\/p>\n<p>Note that Google strives to check and index pages in a reasonably timely manner. For most<br \/>\n  sites, this is three days or more. Don&#8217;t expect Google to index pages the same day that you<br \/>\n  publish them unless you are a news site or have other high-value, extremely time-sensitive<br \/>\n  content.<\/p>\n<p><b>Diagnosing:<\/b><\/p>\n<p>Examine your site logs to see when specific URLs were crawled by <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/google-crawlers\/\">Googlebot<\/a>.<\/p>\n<p>To learn the indexing date, use the URL Inspection tool or do a search for URLs that<br \/>\n  you updated.<\/p>\n<p><b>Treating:<\/b><\/p>\n<p><span class=\"compare-yes\" aria-hidden=\"true\"><\/span><b>Do:<\/b><\/p>\n<ul>\n<li>Use a <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/news-sitemaps\/\">news sitemap<\/a> if your site<br \/>\n  has news content.<\/li>\n<li>Use the <code>&lt;lastmod&gt;<\/code> tag in sitemaps to indicate when an indexed URL has<br \/>\n  been updated.<\/li>\n<li>Use a <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/url-structure\/\">crawlable URL structure<\/a> to help Google find your pages.<\/li>\n<li>Provide standard, <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/internal-links\/#crawlable-links\">crawlable <code>&lt;a&gt;<\/code> links<\/a><br \/>\n  to help Google find your pages.<\/li>\n<li>If your site uses separate HTML for mobile and desktop versions, provide the same set of links<br \/>\n  on the mobile version as you have on the desktop version. If it&#8217;s not possible to provide the<br \/>\n  same set of links on the mobile version, ensure that they&#8217;re included in a<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/sitemaps-overview\/\">sitemap<\/a> file.<br \/>\n  Google only<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/mobile-site-and-mobile-first-indexing-best-practices\/\">indexes the<br \/>\n    mobile version<\/a> of pages, and limiting the links shown there<br \/>\n    can slow down discovery of new pages.<\/li>\n<\/ul>\n<p><span class=\"compare-no\" aria-hidden=\"true\"><\/span><b>Avoid:<\/b><\/p>\n<ul>\n<li>Submitting the same, unchanged sitemap multiple times per day.<\/li>\n<li>Expecting that Googlebot will crawl everything in a sitemap, or crawl them immediately.<br \/>\n  Sitemaps are useful suggestions to Googlebot, not absolute requirements.<\/li>\n<li>Including URLs in your sitemaps that <a href=\"#hide_urls\">you don&#8217;t want to appear in Search<\/a>.<br \/>\n  This can waste your crawl budget on pages that you don&#8217;t want indexed.<\/li>\n<\/ul>\n<h2 id=\"improve_crawl_efficiency\" tabindex=\"-1\">Improve your site&#8217;s crawl efficiency<\/h2>\n<h3 id=\"increase-your-page-loading-speed\" tabindex=\"-1\">Increase your page loading speed<\/h3>\n<p>\n  Google&#8217;s crawling is limited by bandwidth, time, and availability of Googlebot instances.<br \/>\n  If your server responds to requests quicker, we might be able to crawl more pages on your<br \/>\n  site. That said, Google only wants to crawl high quality content, so just making low<br \/>\n  quality pages faster won&#8217;t encourage Googlebot to crawl more of your site; conversely, if we<br \/>\n  think that we&#8217;re missing high-quality content on your site, we&#8217;ll probably increase your<br \/>\n  budget to crawl that content.\n<\/p>\n<p>Here&#8217;s how you can optimize your pages and resources for crawling:<\/p>\n<ul>\n<li>Prevent large but unimportant resources from being loaded by Googlebot using robots.txt.<br \/>\n  Be sure to block only non-critical resources&mdash;that is, resources that aren&#8217;t important to<br \/>\n  understanding the meaning of the page (such as decorative images).<\/li>\n<li>Make sure that your pages are fast to load.<\/li>\n<li>Watch out for long redirect chains, which have a negative effect on crawling.<\/li>\n<li>Both the time to respond to server requests, as well as the time needed to render pages,<br \/>\n  matters, including load and run time for embedded resources such as images and scripts. Be<br \/>\n  aware of large or slow resources required for indexing.<\/li>\n<\/ul>\n<h3 id=\"if-modified-since\" tabindex=\"-1\">Specify content changes with HTTP status codes<\/h3>\n<p>\n  Google generally supports the<br \/>\n  <a href=\"https:\/\/developer.mozilla.org\/docs\/Web\/HTTP\/Headers\/If-Modified-Since\" class=\"external-link\"><code>If-Modified-Since<\/code> and <code>If-None-Match<\/code> HTTP request headers<\/a><br \/>\n  for crawling. Google&#8217;s crawlers don&#8217;t send the headers with all crawl attempts; it depends on<br \/>\n  the use case of the request (for example,<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/google-crawlers\/#adsbot\">AdsBot<\/a> is more<br \/>\n  likely to set the <code>If-Modified-Since<\/code> and <code>If-None-Match<\/code> HTTP request<br \/>\n  headers). If our crawlers send the <code>If-Modified-Since<\/code> header, the header&#8217;s value<br \/>\n  is the <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110#name-if-modified-since\" class=\"external-link\">date and time<\/a><br \/>\n  the content was last crawled. Based on that value, the server may choose to return a<br \/>\n  <code>304 (Not Modified)<\/code> HTTP status code with no response body, in which case Google<br \/>\n  will reuse the content version it crawled the last time. If the content is newer than the date<br \/>\n  specified by the crawler in the <code>If-Modified-Since<\/code> header, the server can return a<br \/>\n  <code>200 (OK)<\/code> HTTP status code with the response body.\n<\/p>\n<p>\n  Independently of the request headers, you can send a <code>304 (Not Modified)<\/code> HTTP<br \/>\n  status code and no response body for any Googlebot request if the content hasn&#8217;t changed since<br \/>\n  Googlebot last visited the URL. This will save your server processing time and resources,<br \/>\n  which may indirectly improve crawl efficiency.\n<\/p>\n<h2 id=\"hide_urls\" tabindex=\"-1\">Hide URLs that you don&#8217;t want in search results<\/h2>\n<p>\n  Wasting server resources on unnecessary pages can reduce crawl activity from pages that are<br \/>\n  important to you, which may cause a significant delay in discovering great new or updated<br \/>\n  content on a site.\n<\/p>\n<aside class=\"key-point\">\n  Blocking or hiding already crawled pages from recrawls won&#8217;t shift your crawl budget to<br \/>\n  another part of your site unless Google is already hitting your site&#8217;s serving limits.<br \/>\n<\/aside>\n<p>\n  Exposing many URLs on your site that you don&#8217;t want crawled by Search can negatively affect a<br \/>\n  site&#8217;s crawling and indexing. Typically these URLs fall into the following categories:\n<\/p>\n<ul>\n<li>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/managing-crawling-of-faceted-navigation-urls\/\">Faceted navigation<\/a><br \/>\n  and <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/google-duplicate-content-caused-by-url-parameters-and-you\/\">session identifiers<\/a>:<br \/>\n  Faceted navigation is typically duplicate content from the site; session identifiers and<br \/>\n  other URL parameters that simply sort or filter the page don&#8217;t provide new content. Learn how to<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/managing-crawling-of-faceted-navigation-urls\/\">manage crawling of faceted navigation pages<\/a>.\n<\/li>\n<li><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/specify-canonical\/\">Duplicate content<\/a>:<br \/>\n  Help Google identify duplicate content to avoid unnecessary crawling.<\/li>\n<li><a href=\"#soft-404-errors\"><code>soft 404<\/code> pages<\/a>: Return a <code>404<\/code><br \/>\n  code when a page no longer exists.<\/li>\n<li><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/spam-policies\/\">Hacked pages<\/a>: Be sure to check<br \/>\n  the <a href=\"https:\/\/search.google.com\/search-console\/security-issues\" class=\"external-link\">Security<br \/>\n  Issues report<\/a> and fix or remove any hacked pages you find.<\/li>\n<li><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/to-infinity-and-beyond-no\/\">Infinite spaces<\/a> and proxies:<br \/>\n  Block these from crawling with robots.txt.<\/li>\n<li><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/spam-policies\/\">Low quality and spam content<\/a>:<br \/>\n  Good to avoid, obviously.<\/li>\n<li>Shopping cart pages, infinite scrolling pages, and pages that perform an action (such as<br \/>\n  &#8220;sign up&#8221; or &#8220;buy now&#8221; pages).<\/li>\n<\/ul>\n<p><span class=\"compare-yes\" aria-hidden=\"true\"><\/span><b>Do:<\/b><\/p>\n<ul>\n<li>Use robots.txt if you don&#8217;t want Google to crawl a resource or page at all.<\/li>\n<li>If a common resource is reused on multiple pages (such as a shared image or JavaScript<br \/>\n  file), reference the resource from the same URL in each page, so that Google can cache and<br \/>\n  reuse the same resource without needing to request the same resource multiple times.<\/li>\n<\/ul>\n<p><span class=\"compare-no\" aria-hidden=\"true\"><\/span><b>Avoid:<\/b><\/p>\n<ul>\n<li>Don&#8217;t add or remove pages or directories from robots.txt regularly as a way of reallocating<br \/>\n  crawl budget for your site. Use robots.txt only for pages or resources that<br \/>\n  you don&#8217;t want to appear on Google for the long run.<\/li>\n<li>Don&#8217;t rotate sitemaps or use other temporary hiding mechanisms to reallocate budget.<\/li>\n<\/ul>\n<h3 id=\"soft-404-errors\" tabindex=\"-1\"><code>soft 404<\/code> errors<\/h3>\n<p>\n  A <code>soft 404<\/code> error is when a URL that returns a page telling the user that the page does<br \/>\n  not exist <b>and also a<br \/>\n  <a href=\"https:\/\/en.wikipedia.org\/wiki\/List_of_HTTP_status_codes#2xx_Success\" class=\"external-link\"><code>200 (success)<\/code><\/a><br \/>\n  status code<\/b>. In some cases, it might be a page with no main content or empty page.\n<\/p>\n<p>\n  Such pages may be generated for various reasons by your website&#8217;s web server or content<br \/>\n  management system, or the user&#8217;s browser. For example:\n<\/p>\n<ul>\n<li>A missing server-side include file.<\/li>\n<li>A broken connection to the database.<\/li>\n<li>An empty internal search result page.<\/li>\n<li>An unloaded or otherwise missing JavaScript file.<\/li>\n<\/ul>\n<p>\n  It&#8217;s a bad user experience to return a <code>200 (success)<\/code> status code, but then<br \/>\n  display or suggest an error message or some kind of error on the page. Users may think the<br \/>\n  page is a live working page, but then are presented with some kind of error. Such pages are<br \/>\n  excluded from Search.\n<\/p>\n<p>\n  When Google&#8217;s algorithms detect that the page is actually an error page based on its content,<br \/>\n  Search Console will show a <code>soft 404<\/code> error in the site&#8217;s<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203\" class=\"external-link\">Page Indexing report<\/a>.\n<\/p>\n<h4 id=\"fix-soft-404-errors\" tabindex=\"-1\">Fix <code>soft 404<\/code> errors<\/h4>\n<p>\n  Depending on the state of the page and the outcome you want, you can solve <code>soft 404<\/code><br \/>\n  errors in multiple ways:\n<\/p>\n<ul>\n<li><a href=\"#pagegone\">The page and content are no longer available.<\/a><\/li>\n<li><a href=\"#pagemoved\">The page or content is now somewhere else.<\/a><\/li>\n<li><a href=\"#pageother\">The page and content still exist.<\/a><\/li>\n<\/ul>\n<p>\n  Try to determine which solution would be the best for your users.\n<\/p>\n<h5 id=\"pagegone\" tabindex=\"-1\">The page and content are no longer available<\/h5>\n<p>\n  If you removed the page and there&#8217;s no replacement page on your site with similar content,<br \/>\n  return a<br \/>\n  <a href=\"https:\/\/en.wikipedia.org\/wiki\/List_of_HTTP_status_codes#4xx_Client_errors\" class=\"external-link\"><code>404 (not found)<\/code> o <code>410 (gone)<\/code><\/a><br \/>\n  response (status) code for the page. These status codes indicate to search engines that the<br \/>\n  page doesn&#8217;t exist and you don&#8217;t want search engines to index the page.\n<\/p>\n<p>\n  If you have access to your server&#8217;s configuration files, you can make these error pages useful<br \/>\n  to users by customizing them. A good custom <code>404<\/code> page helps people find the<br \/>\n  information they&#8217;re looking for, and also provides other helpful content that encourages<br \/>\n  people to explore your site further. Here are some tips for designing a useful custom<br \/>\n  <code>404<\/code> page:\n<\/p>\n<ul>\n<li>\n    Tell visitors clearly that the page they&#8217;re looking for can&#8217;t be found. Use language that is<br \/>\n    friendly and inviting.<\/li>\n<li>\n    Make sure your <code>404<\/code> page has the same look and feel (including navigation) as<br \/>\n    the rest of your site.<\/li>\n<li>\n    Consider adding links to your most popular articles or posts, as well as a link to your<br \/>\n    site&#8217;s home page.<\/li>\n<li>Think about providing a way for users to report a broken link.<\/li>\n<\/ul>\n<p>\n  Custom <code>404<\/code> pages are created solely for users. Since these pages are useless from<br \/>\n  a search engine&#8217;s perspective, make sure the server returns a <code>404<\/code> HTTP status<br \/>\n  code to prevent having the pages indexed.\n<\/p>\n<h5 id=\"pagemoved\" tabindex=\"-1\">The page or content is now somewhere else<\/h5>\n<p>\n  If your page has moved or has a clear replacement on your site, return a<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/redirects-and-google\/\"><code>301 (permanent redirect)<\/code><\/a><br \/>\n  to redirect the user. This won&#8217;t interrupt their browsing experience and it&#8217;s also a great<br \/>\n  way to tell search engines about the new location of the page. Use the<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9012289\" class=\"external-link\">URL Inspection tool<\/a><br \/>\n  to verify whether your URL is actually returning the correct code.\n<\/p>\n<h5 id=\"pageother\" tabindex=\"-1\">The page and content still exist<\/h5>\n<p>\n  If an otherwise good page was flagged with a <code>soft 404<\/code> error, it&#8217;s likely it<br \/>\n  didn&#8217;t load properly for Googlebot, it was missing critical resources, or it displayed a<br \/>\n  prominent error message during rendering. Use the<br \/>\n  <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9012289\" class=\"external-link\">URL Inspection tool<\/a><br \/>\n  to examine the rendered content and the returned HTTP code. If the rendered page is blank,<br \/>\n  nearly blank, or the content has an error message, it could be that your page references many<br \/>\n  resources that can&#8217;t be loaded (images, scripts, and other non-textual elements), which can be<br \/>\n  interpreted as a <code>soft 404<\/code>.<br \/>\n  Reasons that resources can&#8217;t be loaded include blocked resources (blocked by<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/\">robots.txt<\/a>), having too many<br \/>\n  resources on a page, various server errors, or slow loading or very large resources.\n<\/p>\n<h2 id=\"emergencies\" tabindex=\"-1\">Handle overcrawling of your site (emergencies)<\/h2>\n<p>Googlebot has algorithms to prevent it from overwhelming your site with crawl requests.<br \/>\n  However, if you find that Googlebot is overwhelming your site, there are a few things you can<br \/>\n  do.<\/p>\n<p><b>Diagnosing<\/b>:<\/p>\n<p>Monitor your server for excessive Googlebot requests to your site.<\/p>\n<p><b>Treating<\/b>:<\/p>\n<p>In an emergency, we recommend the following steps to slow down an overwhelming crawl from<br \/>\n  Googlebot:<\/p>\n<ol>\n<li>Return <code>503<\/code> o <code>429<\/code> HTTP response status codes <i>temporarily<\/i> for Googlebot requests when your<br \/>\n  server is overloaded. Googlebot will retry these URLs for about 2 days. Note that returning<br \/>\n  &#8220;no availability&#8221; codes for more than a few days will cause Google to permanently slow or<br \/>\n  stop crawling URLs on your site, so follow the additional next steps.<\/li>\n<li>\n  When the crawl rate goes down, stop returning <code>503<\/code> o <code>429<\/code> HTTP<br \/>\n  response status codes for crawl requests; returning <code>503<\/code> o <code>429<\/code> for<br \/>\n  more than 2 days will cause Google to drop those URLs from the index.\n<\/li>\n<li>Monitor your crawling and your host capacity over time.<\/li>\n<li name=\"adsbot\">\n  If the problematic crawler is one of the<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-googles-special-case-crawlers\/#adsbot-mobile-web\">AdsBot crawlers<\/a>,<br \/>\n  the problem is likely that you have created<br \/>\n  <a href=\"https:\/\/support.google.com\/google-ads\/answer\/2497706\" class=\"external-link\">Dynamic Search Ad targets<\/a><br \/>\n  for your site that Google is trying to crawl. This crawl will reoccur every 3 weeks. If you don&#8217;t<br \/>\n  have the server capacity to handle these crawls, either limit your ad targets or get increased<br \/>\n  serving capacity.\n<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to troubleshoot crawling errors.<\/p>\n","protected":false},"menu_order":26,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[684],"ar_kb_etiket":[],"class_list":["post-25582","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-crawler-management-crawling-and-indexing"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25582","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25582\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27592"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25582"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25582"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25582"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}