{"id":16562,"date":"2026-09-02T11:35:46","date_gmt":"2026-09-02T08:35:46","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/google-crawlers\/"},"modified":"2026-09-02T16:46:02","modified_gmt":"2026-09-02T13:46:02","slug":"google-crawlers","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/google-crawlers\/","title":{"rendered":"Google Crawlers and How to Verify Them"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Google runs several crawlers. The main ones are Googlebot Smartphone (the primary crawler for indexing), Googlebot Desktop, Googlebot Image, Googlebot Video, and separate agents for AdsBot and user-triggered fetches.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Verification matters<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">User-agent strings are trivially forged. Scrapers routinely claim to be Googlebot to bypass rate limits. Verify with a reverse DNS lookup on the requesting IP, it must resolve to <code>googlebot.com<\/code> o <code>google.com<\/code>, then a forward lookup back to the same IP. Google also publishes its IP ranges as JSON.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Crawl rate<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Google adjusts crawl rate automatically based on server response. Consistent 5xx errors or slow responses cause it to back off. There is no longer a manual crawl rate setting in Search Console; if you need to slow it urgently, return 503 or 429, but only briefly, because sustained 5xx responses lead to de-indexing.<\/p>\n\n\n\n<div class=\"ar-kb-tavsiye\"><p><strong>Alien Road recommends:<\/strong> verify before you block. We have been called in more than once to fix &#8220;Googlebot is hammering the server&#8221;, and found a bot-protection rule that was blocking real Googlebot because someone matched on user agent alone. Verify by DNS, allow the real one, and rate-limit the impostors.<\/p><\/div>\n\n\n","protected":false},"excerpt":{"rendered":"<p>Which user agents Google operates, and how to confirm a request claiming to be Googlebot really is.<\/p>\n","protected":false},"menu_order":10,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[502],"ar_kb_etiket":[532,564,565],"class_list":["post-16562","ar_kb","type-ar_kb","status-publish","hentry","ar_kb_kategori-crawler-management","ar_kb_etiket-crawling","ar_kb_etiket-googlebot","ar_kb_etiket-verification"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16562","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16562\/revisions"}],"predecessor-version":[{"id":23695,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/16562\/revisions\/23695"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=16562"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=16562"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=16562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}