{"id":25557,"date":"2026-09-05T02:45:37","date_gmt":"2026-09-04T23:45:37","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/googlebot\/"},"modified":"2026-09-05T02:45:37","modified_gmt":"2026-09-04T23:45:37","slug":"googlebot","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/googlebot\/","title":{"rendered":"Googlebot"},"content":{"rendered":"<p>\n  Googlebot is the generic name for two types of<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-search-works\/\">web crawlers<\/a> used by Google Search:\n<\/p>\n<ul>\n<li>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-googles-common-crawlers\/#googlebot-smartphone\"><b>Googlebot Smartphone<\/b><\/a>:<br \/>\n    a mobile crawler that simulates a user on a mobile device.\n  <\/li>\n<li>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-googles-common-crawlers\/#googlebot-desktop\"><b>Googlebot Desktop<\/b><\/a>:<br \/>\n    a desktop crawler that simulates a user on desktop.\n  <\/li>\n<\/ul>\n<p>\n  You can identify the subtype of Googlebot by looking at the<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/google-crawlers\/\">HTTP <code>user-agent<\/code> request header<\/a><br \/>\n  in the request. However, both crawler types obey the same product token (user agent token) in<br \/>\n  robots.txt, and so you cannot selectively target either Googlebot Smartphone or Googlebot<br \/>\n  Desktop using robots.txt.\n<\/p>\n<p>\n  For most sites Google Search primarily<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/mobile-site-and-mobile-first-indexing-best-practices\/\">indexes the mobile version<\/a><br \/>\n  of the content. As such the majority of Googlebot crawl requests will be made using the mobile<br \/>\n  crawler, and a minority using the desktop crawler.\n<\/p>\n<h2 id=\"how-googlebot-accesses-your-site\" tabindex=\"-1\">How Googlebot accesses your site<\/h2>\n<p>\n  For most sites, Googlebot shouldn&#8217;t access your site more than once every few seconds on<br \/>\n  average. However, due to delays it&#8217;s possible that the rate will appear to be slightly higher<br \/>\n  over short periods. If your site is having trouble keeping up with Google&#8217;s crawling requests, you<br \/>\n  can <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/reduce-the-google-crawl-rate\/\">reduce the crawl rate.<\/a>\n<\/p>\n<p>\n  When crawling for Google Search, Googlebot crawls the first 2MB of a<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/file-types-indexed\/\">supported file type<\/a>, and<br \/>\n  the first 64MB of a PDF file. From a rendering perspective, each resource referenced in the HTML<br \/>\n  (such as CSS and JavaScript) is fetched separately, and each resource fetch is bound by the same<br \/>\n  file size limit that applies to other files (except PDF files).<br \/>\n  Once the cutoff limit is reached, Googlebot stops the fetch and only sends the already downloaded<br \/>\n  part of the file for indexing consideration. The file size limit is applied on the uncompressed<br \/>\n  data.<br \/>\n  Other Google crawlers, for example Googlebot Video and Googlebot Image, may have<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/overview-of-google-crawlers-and-fetchers-user-agents\/#file-size-limits\" class=\"external-link\">different limits<\/a>.\n<\/p>\n<p id=\"timezone\">\n  When crawling from IP addresses in the US, the timezone of Googlebot is<br \/>\n  <a href=\"https:\/\/g.co\/kgs\/WSf8oR\" class=\"external-link\">Pacific Time<\/a>.\n<\/p>\n<p>\n  Other<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/google-crawlers\/#crawl-technical-props\">technical properties of Googlebot<\/a><br \/>\n  are described in the overview of Google&#8217;s crawlers.\n<\/p>\n<h2 id=\"blocking-googlebot-from-visiting-your-site\" tabindex=\"-1\">Blocking Googlebot from visiting your site<\/h2>\n<p>\n  Googlebot discovers new URLs to crawl primarily from links embedded in previously crawled pages.<br \/>\n  It&#8217;s almost impossible to keep a site secret by not publishing links to it. For example, as soon<br \/>\n  as someone clicks a link from your &#8220;secret&#8221; site to another site, your &#8220;secret&#8221; site URL may<br \/>\n  appear in the referrer tag and can be stored and published by the other site in its referrer log.\n<\/p>\n<p>\n  If you want to prevent Googlebot from crawling content on your site, you have a<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/control-what-you-share-with-google\/\">number of options<\/a>. Remember<br \/>\n  there&#8217;s a difference between <i>crawling<\/i> and <i>indexing<\/i>; blocking Googlebot from crawling<br \/>\n  a page doesn&#8217;t prevent the URL of the page from appearing in search results:\n<\/p>\n<ul>\n<li>\n    <b>Prevent Googlebot from crawling a page?<\/b> Use a<br \/>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/robots-txt-intro\/\">robots.txt file<\/a>.\n  <\/li>\n<li>\n    <b>Don&#8217;t want Google to index a page?<\/b> Use<br \/>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/block-search-indexing-with-noindex\/\"><code>noindex<\/code><\/a>.\n  <\/li>\n<li>\n    <b>Prevent a page from being accessible at all by both crawlers or users?<\/b> Use<br \/>\n    <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/control-what-you-share-with-google\/\">another method, such as password protection<\/a>.\n  <\/li>\n<\/ul>\n<p>\n  Blocking Googlebot affects Google Search (including Discover and all Google Search features), as<br \/>\n  well as other products such as Google Images, Google Video, and Google News.\n<\/p>\n<h2 id=\"verifying-googlebot\" tabindex=\"-1\">Verifying Googlebot<\/h2>\n<p>\n  Before you decide to block Googlebot, be aware that the HTTP <code>user-agent<\/code> request<br \/>\n  header used by Googlebot is often spoofed by other crawlers. It&#8217;s important to verify that a<br \/>\n  problematic request actually comes from Google. The best way to verify that a request actually<br \/>\n  comes from Googlebot is to<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/verify-requests-from-google-crawlers-and-fetchers\/#manual\">use a reverse DNS lookup<\/a><br \/>\n  on the source IP of the request, or to match the source IP against the<br \/>\n  <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/verify-requests-from-google-crawlers-and-fetchers\/#use-automatic-solutions\">Googlebot IP ranges<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Googlebot is the generic name of the web crawler used by Google Search. Discover what Googlebot is, how it accesses your site, and how to block Googlebot.<\/p>\n","protected":false},"menu_order":27,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[689],"ar_kb_etiket":[],"class_list":["post-25557","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-google-crawlers"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25557","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25557\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27580"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25557"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25557"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25557"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}