Tuesday, March 31, 2026<\/p>\n
\n If you tuned into \n For a long time, the name “Googlebot” has conjured up the image of a single, \n Let’s clear up a historical misnomer first. Back in the early 2000s, Google \n When you see Googlebot in your server logs, you are just looking at Google \n This is where things get somewhat confusing. Every client of the crawler \n Googlebot currently fetches up to 2MB for any individual URL (excluding \n Image and video crawlers typically have a wide range of threshold values, \n For any other crawler that doesn’t specify a limit, the default is 15MB What does this mean for the bytes your server sends over the wire?<\/p>\n \n For the vast majority of the web, a 2MB HTML payload is massive, and you \n Once the crawler has successfully retrieved the bytes (up to the limit), it \n However, remember that the WRS can only execute the code that the crawler \n To ensure Googlebot can efficiently fetch and understand your content, keep \n Note that this limit is not set in stone<\/strong> and may change \n Crawling isn’t magic; it’s a highly orchestrated, scaled exchange of bytes. Happy optimizing!<\/p>\n \n Want to hear more behind-the-scenes details? Check out
\n episode 105 of the Search Off the Record podcast<\/a>,
\n you might have heard us diving deep into a topic that is close to our hearts
\n (and our servers): the inner workings of Googlebot.\n<\/p>\n
\n tireless robot systematically reading the internet. But the reality is a bit
\n more complex — and a lot more interesting. Today, we want to pop the
\n hood on our crawling infrastructure, with a special focus on the very thing
\n that makes our own heads spin: bytesize limits.\n<\/p>\nFirst, Googlebot isn’t a single program<\/h2>\n
\n had one product, so we had one crawler. The name “Googlebot” stuck. But
\n today, Googlebot is just a user of something that resembles a centralized
\n crawling platform.\n<\/p>\n
\n Search. Dozens of other clients — Google Shopping, AdSense, and
\n more — all route their crawl requests through this same underlying
\n infrastructure under different crawler names, the larger ones documented
\n on the
\n Google Crawler infrastructure site<\/a>.\n<\/p>\nThe 2MB limit: what happens to your bytes?<\/h2>\n
\n infrastructure needs to set some settings for their fetches. These settings
\n include the user agent string, what user agent tokens will they look for in
\n robots.txt, and how many bytes they will fetch from a single URL.\n<\/p>\n
\n PDFs). This means it crawls only the first 2MB of a resource, including the
\n HTTP header. For PDF files, the limit is 64MB.\n<\/p>\n
\n and it largely depends on the product that they’re fetching for. For
\n example, fetching a favicon might have a very low limit, unlike Image Search.\n<\/p>\n
\n regardless of content type.\n<\/p>\n\n
\n Googlebot doesn’t reject the page. Instead, it stops the fetch exactly at
\n the 2MB cutoff. Note that the limit includes HTTP request headers.\n <\/li>\n
\n 2MB of bytes) is passed along to our indexing systems and the Web
\n Rendering Service (WRS) as if it were the complete file.\n <\/li>\n
\n 2MB threshold are entirely ignored. They aren’t fetched, they aren’t
\n rendered, and they aren’t indexed.\n <\/li>\n
\n HTML (excluding media, fonts, and a few exotic files) will be fetched by
\n WRS with Googlebot like the parent HTML. They have their own, separate,
\n per-URL byte counter and don’t count towards the size of the parent page.\n <\/li>\n<\/ol>\n
\n will never hit this limit. However, if your page includes bloated inline
\n base64 images, massive blocks of inline CSS\/JavaScript, or starts with
\n megabytes of menus, you could accidentally push your actual textual content
\n or critical structured data past the 2MB mark. If those crucial bytes aren’t
\n fetched, to Googlebot, they simply don’t exist.\n<\/p>\nRendering the bytes<\/h2>\n
\n passes the baton to the WRS. The WRS processes JavaScript and executes
\n client-side code similar to a modern browser to understand the final visual
\n and textual state of the page. Rendering pulls in and executes JavaScript and
\n CSS files, and processes XHR requests to better understand the page’s textual
\n content and structure (it doesn’t request images or videos). For each
\n requested resource, the 2MB limit also applies.\n<\/p>\n
\n actually retrieved. Furthermore, the
\n WRS operates statelessly<\/a>
\n — it clears local storage and session data between requests. This may
\n have particular implications for how dynamic, JavaScript-dependent elements
\n are interpreted by our systems.\n<\/p>\nBest practices for your bytes<\/h2>\n
\n these byte-level best practices in mind:\n<\/p>\n\n
\n external files. While the initial HTML document is capped at 2MB, external
\n scripts, and stylesheets are fetched separately (subject to their own
\n limits).\n <\/li>\n
\n like meta tags, <title><\/code> elements,
\n <link><\/code> elements, canonicals, and essential structured
\n data — higher up in the HTML document. This ensures they are unlikely
\n to be found below the cutoff.\n <\/li>\n
\n response times. If your server is struggling to serve bytes, our crawlers
\n will automatically back off to avoid overloading your infrastructure, which
\n will drop your crawl frequency.\n <\/li>\n<\/ul>\n
\n over time as the web evolves and HTML pages grow in size. (Or shrink.
\n Hopefully shrink.)\n<\/p>\n
\n By understanding how our central fetching infrastructure retrieves and
\n limits those bytes, you can ensure your site’s most important content always
\n makes the cut.\n<\/p>\n
\n Episode 105 of the Search Off the Record podcast on YouTube<\/a>
\n or wherever you get your podcasts!<\/em>\n<\/p>\n