{"id":25334,"date":"2024-12-03T00:00:00","date_gmt":"2024-12-03T00:00:00","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/crawling-december-the-how-and-why-of-googlebot-crawling\/"},"modified":"2024-12-03T00:00:00","modified_gmt":"2024-12-03T00:00:00","slug":"crawling-december-the-how-and-why-of-googlebot-crawling","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/crawling-december-the-how-and-why-of-googlebot-crawling\/","title":{"rendered":"Crawling December: The how and why of Googlebot crawling"},"content":{"rendered":"

Tuesday, December 3, 2024<\/p>\n

\n You may have heard that Google Search needs to do a bit of work before a web page can show up in
\n Google Search results. One of these steps is called crawling. Crawling for Google Search is done
\n by Googlebot, a program running on Google servers that retrieves a URL and handles things like
\n network errors, redirects, and other small complications that it might encounter as it works its
\n way through the web. But there are a few details that aren’t often talked about. Each week this
\n month we’re going to explore some of those details as they may have a significant effect on how
\n your sites are crawled.\n<\/p>\n

Backing up a little: What is crawling?<\/h2>\n

\n Crawling<\/a> is the process of
\n discovering new and revisiting updated web pages, and downloading them. In short, Googlebot gets
\n a URL, makes an HTTP request to the server hosting it and then deals with the response from that
\n server, possibly following redirects, handling errors, and passing the page content on to
\n Google’s indexing system.\n<\/p>\n

\n But modern web pages aren’t just pure HTML, so what about the other resources that make up a page?
\n How does crawling these resources affect “crawl budget”? Are these resources cacheable on Google’s
\n side? And is there a difference between URLs that have not been crawled before and those that are
\n already indexed? In this post we’re going to answer these questions, and more!\n<\/p>\n

Googlebot and crawling page resources<\/h2>\n

\n Beyond HTML, modern websites use a combination of different technologies such as JavaScript and
\n CSS to offer users vibrant experiences and useful functionalities. When accessing such pages with
\n a browser, the browser first downloads the parent URL which hosts the data needed to start
\n building the page for the user — the HTML of the page. This initial data may contain
\n references to resources like JavaScript and CSS, but also images and videos that the browser will
\n once again download to eventually construct the final page which is then presented to the user.\n<\/p>\n

\n Google does exactly the same thing<\/a>,
\n though slightly differently:\n<\/p>\n

    \n
  1. Googlebot downloads the initial data from the parent URL — the HTML of the page.<\/li>\n
  2. Googlebot passes on the fetched data to the Web Rendering Service (WRS).<\/li>\n
  3. Using Googlebot, WRS downloads the resources referenced in the original data.<\/li>\n
  4. WRS constructs the page using all the downloaded resources as a user’s browser would.<\/li>\n<\/ol>\n

    \n Compared to a browser, the time between each step may be significantly longer due to scheduling
    \n constraints such as the perceived load of the server hosting the resources needed for rendering a
    \n page. And this is where
    \n     crawl budget<\/a>
    \n slips into the conversation.\n<\/p>\n

    \n Crawling the resources needed to render a page will chip away from the crawl budget of the
    \n hostname that’s hosting the resource. To ameliorate this, WRS attempts to cache every resource
    \n (JavaScript and CSS) referenced in pages it renders. The time to live of the WRS cache is
    \n unaffected by HTTP caching directives; instead WRS caches everything for up to 30 days, which
    \n helps preserve the site’s crawl budget for other crawl tasks.\n<\/p>\n

    \n From the site owners’ perspective, managing how and what resources are crawled can influence the
    \n site’s crawl budget; we recommend:\n<\/p>\n

      \n
    1. \n Use as few resources as feasible<\/strong> to offer users a great experience; the fewer
      \n resources are needed for rendering a page, the less crawl budget is spent during rendering.\n <\/li>\n
    2. \n Use cache-busting parameters cautiously<\/strong>: if the URLs of resources change,
      \n Google may need to crawl the resources again, even if their contents haven’t changed. This, of
      \n course, will consume crawl budget.\n <\/li>\n
    3. \n Host resources on a different hostname<\/strong> from the main site, for example by
      \n employing a CDN or just hosting the resources on a different subdomain. This will shift crawl
      \n budget concerns to the host that’s serving the resources.<\/p>\n