\n This page details the latest major updates made to Google’s crawling documentation.\n<\/p>\n
\n To get the latest crawling documentation updates delivered to you, add the URL of this \n What<\/b>: Polished and clarified the \n Why<\/b>: To make the documentation easier to understand and ensure technical terms \n What<\/b>: Updated the NotebookLM user agent to be \n Why<\/b>: NotebookLM is now Gemini Notebook<\/a>.\n <\/p>\n<\/dd>\n \n What<\/b>: Updated the \n Why<\/b>: The user agent string for \n What<\/b>: Added documentation on how to authenticate requests with Web Bot Auth<\/a>.\n <\/p>\n \n Why<\/b>: To provide site owners with instructions on how to verify requests with Web Bot \n What<\/b>: Added the Google-Agent user agent<\/a> and added the IP ranges for user triggered \n Why<\/b>: The Google-Agent user agent is rolling out over the next few weeks, and will be \n What<\/b>: Added a new page about how Google’s crawling works<\/a>.\n <\/p>\n \n Why<\/b>: Based on questions we’ve received over the years, we’ve put together a resource \n What<\/b>: Updated the location of Google’s \n Why<\/b>: Their previous location in \n What<\/b>: Moved the information about the default file size limits of Google’s crawlers and \n Why<\/b>: The original location of this information was not the most logical place as it \n What<\/b>: Added the Google Messages<\/a> fetcher to the list of user-triggered fetchers.\n <\/p>\n \n Why<\/b>: To help site owners identify traffic from Google Messages when it generates link previews for URLs sent in chat messages.\n <\/p>\n<\/dd>\n<\/dl>\n \n What<\/b>: Migrated the following documentation to \n Why<\/b>: This documentation is more relevant to many Google products that use Google’s crawlers, \n What<\/b>: Updated the \n Why<\/b>: We received feedback through the help community that more information would be helpful.\n <\/p>\n<\/dd>\n<\/dl>\n \n What<\/b>: Migrated crawling documentation from the \n Why<\/b>: Google’s crawling infrastructure is shared across a variety of Google products beyond Search, \n What<\/b>: We added the \n Why<\/b>: The \n What<\/b>: Based on feedback, we added the \n What<\/b>: Update the documentation on Google Read Aloud<\/a> with an updated list of Google products that use the Read Aloud service.\n <\/p>\n \n Why<\/b>: Other Google products can now use the Google Read Aloud service.\n <\/p>\n<\/dd>\n \n What<\/b>: Based on feedback, we added \n What<\/b>: Updated Google Read Aloud user agent<\/a> \n Why<\/b>: To accommodate sites which don’t support old browser versions.\n <\/p>\n<\/dd>\n<\/dl>\n \n What<\/b>: Based on publisher feedback, we updated the Google-Extended<\/a>
\n page to your feed reader<\/a>,
\n or add the feed URL directly:
\n https:\/\/developers.google.com\/crawling\/docs\/changelog\/crawling_docs_updates.rss<\/code>.\n<\/p>\nJuly 2026<\/h2>\n
\n
Polished and clarified the crawl budget guide<\/h3>\n
\n Optimize your crawl budget<\/a> page to improve clarity,
\n terminology consistency, and flow.\n <\/p>\n
\n (like crawl capacity limit<\/i>) are used consistently throughout.\n <\/p>\n<\/dd>\nUpdating the NotebookLM user agent<\/h3>\n
Google-GeminiNotebook<\/code><\/a>.
\n If you hardcoded the old value in your code, update the string to avoid potential bugs. We
\n will continue to support the old value to allow for a smooth transition.\n <\/p>\nCorrected the user agent string for
Google-InspectionTool<\/code> user agent<\/h3>\n
\n Google-InspectionTool<\/code><\/a>
\n user agent string.\n <\/p>\nGoogle-InspectionTool<\/code> mistakenly included
\n a semicolon, which didn’t match the actual user agent string in production. This is now corrected in
\n our documentation.\n <\/p>\n<\/dd>\n<\/dl>\nMay 2026<\/h2>\n
\n
Adding Web Bot Auth documentation<\/h3>\n
\n Auth during the experimental phase, as some Google user agents are now starting
\n to use Web Bot Auth.\n <\/p>\n<\/dd>\n<\/dl>\nMarch 2026<\/h2>\n
\n
Added the Google-Agent user agent<\/h3>\n
\n agents to the documentation on verifying requests from Google crawlers and fetchers<\/a>.\n <\/p>\n
\n used by Google agents hosted on Google infrastructure to navigate the web and perform actions
\n upon user request.\n <\/p>\n<\/dd>\nAdded an overview page about Google’s web crawling<\/h3>\n
\n page with basic educational information about crawling to better highlight various resources
\n about crawling that are available to site owners.\n <\/p>\n<\/dd>\n<\/dl>\nFebruary 2026<\/h2>\n
\n
Updated the location of Google’s IP ranges for common crawlers, special crawlers, and user-triggered fetchers<\/h3>\n
\n IP ranges for common crawlers, special crawlers, and user-triggered fetchers<\/a>
\n to the \/crawling\/ipranges<\/code> directory. The old location will continue to work for
\n the time being, but we recommend updating your links to the new location.\n <\/p>\n\/search\/apis\/ipranges<\/code> was not the most
\n logical place, as these IP ranges are used by many Google products (not just Search).\n <\/p>\n<\/dd>\nAdded information about the default file size limits for Google’s crawlers and fetchers<\/h3>\n
\n fetchers from the
\n Googlebot documentation<\/a>
\n to the
\n crawling documentation<\/a>.\n <\/p>\n
\n applies to all of Google’s crawlers and fetchers, and so we moved it to the crawler
\n infrastructure documentation.\n <\/p>\n<\/dd>\n<\/dl>\nJanuary 2026<\/h2>\n
\n
Added Google Messages to the list of user-triggered fetchers<\/h3>\n
December 2025<\/h2>\n
\n
Migrated more documentation to the Google crawling documentation site<\/h3>\n
\n Google’s crawling infrastructure site<\/a>. The functionality hasn’t changed,
\n only the location of the documentation and some minor wording changes to clarify that some
\n guidance applies to both Google Search and other Google products.\n <\/p>\n\n
\n not just Search.\n <\/p>\n<\/dd>\nUpdated Google Read Aloud user agent page<\/h3>\n
\n Google Read Aloud user agent page<\/a>
\n to provide additional details on how it functions, in particular that it uses stateless
\n rendering, and that it needs to access the page to see the meta tags used on the page.\n <\/p>\nNovember 2025<\/h2>\n
\n
Migrated documentation to the new Google crawling documentation site<\/h3>\n
\n Google Search Central<\/a>
\n to a new Google crawling documentation site<\/a>. The content hasn’t changed,
\n only the location. Notable moves include:\n <\/p>\n\n
\n including Google Shopping, News, Gemini, AdSense, and more. The new site
\n is a more logical home for this documentation and makes it easier to document new features
\n and updates that are relevant to all of these products.\n <\/p>\n<\/dd>\nAdded the
Google-Pinpoint<\/code> fetcher<\/h3>\n
\n Google-Pinpoint<\/code><\/a>
\n fetcher to the list of user-triggered fetchers.\n <\/p>\nGoogle-Pinpoint<\/code> fetcher is used by the Pinpoint research tool.\n <\/p>\n<\/dd>\nAdded a new user-triggered fetcher<\/h3>\n
\n Google-CWS<\/code><\/a>
\n fetcher to the list of user-triggered fetchers.\n <\/p>\n<\/dd>\n<\/dl>\nOctober 2025<\/h2>\n
\n
Updated the list of Google products that use the Read Aloud service<\/h3>\n
Added
Google-NotebookLM<\/code> to the list of user-triggered fetchers<\/h3>\n
\n Google-NotebookLM<\/code><\/a>
\n to the list of user-triggered fetchers.\n <\/p>\n<\/dd>\n<\/dl>\nJuly 2025<\/h2>\n
\n
Updated the Google Read Aloud user agent<\/h3>\n
\n in HTTP requests with newer browser versions.\n <\/p>\nApril 2025<\/h2>\n
\n
Updated the description of the Google-Extended product token<\/h3>\n
\n product token description to provide additional specificity and clarity.\n <\/p>\n<\/dd>\nCorrected the description of the crawler preferences addressed to the
Googlebot-News<\/code> user agent<\/h3>\n