Monday, December 9, 2024<\/p>\n
\nAllow us to cache, pretty please.\n<\/p>\n
\n As the internet grew over the years, so did how much Google crawls. While Google’s crawling
\n infrastructure supports heuristic caching mechanisms, in fact always had, the number of requests
\n that can be returned from local caches has decreased: 10 years ago about 0.026% of the total
\n fetches were cacheable, which is already not that impressive; today that number is 0.017%.\n<\/p>\n
\n Caching is a critical piece of the large puzzle that is the internet. Caching allows pages to load
\n lightning fast on revisits, it saves computing resources and thus also natural resources, and
\n saves a tremendous amount of expensive bandwidth for both the clients and servers.\n<\/p>\n
\n Especially if you have a large site with rarely-changing content under individual URLs, allowing \n We strongly recommend using \n As for what you consider a change that requires clients to refresh their caches, that’s up to you. \n Google’s crawlers support \n Google’s crawlers that support caching will send the \n On the client side, like a user’s browser or Googlebot, the content under that URL is retrieved \n Similarly to \n We have but a couple recommendations if you’re using \n If you’re like me, wrapping my head around how heuristic caching works is challenging, however \n If you’re in the business of making your users happy and perhaps also want to potentially save a \n If you wanna chat about caching, head to your nearest
\n caching locally may help your site be crawled more efficiently. Google’s crawling infrastructure
\n supports heuristic HTTP caching as defined by the
\n HTTP caching standard<\/a>,
\n specifically through the ETag<\/code> response- and If-None-Match<\/code> request
\n header, and the Last-Modified<\/code> response- and If-Modified-Since<\/code> request
\n header.\n<\/p>\nETag<\/code> because it’s less prone to errors and mistakes (the
\n value is not structured unlike the Last-Modified<\/code> value). And, if you have the option,
\n set them both: the internet will thank you. Maybe.\n<\/p>\n
\n Our recommendation is that you require a cache refresh on significant changes to your content; if
\n you only updated the copyright date at the bottom of your page, that’s probably not significant.\n<\/p>\nETag<\/code> and If-None-Match<\/code><\/h2>\nETag<\/code> based conditional requests exactly as defined in the
\n HTTP caching standard<\/a>.
\n That is, to signal caching preference to Google’s crawlers, set the Etag<\/code> value to any
\n arbitrary ASCII string (usually a hash of the content or version number, but it could also be a
\n piece of the \u03c0, up to you) unique to the representation of the content hosted by the accessed URL.
\n For example, if you host different versions of the same content under the same URL (say, mobile
\n and desktop version), each version could have its own unique ETag<\/code> value.\n<\/p>\nETag<\/code> value returned for a
\n previous crawl of that URL in the If-None-Match header<\/code>. If the ETag<\/code>
\n value sent by the crawler matches the current value the server generated, your server should
\n return an HTTP 304<\/code> (Not modified) status code with no HTTP body. This last bit, no
\n HTTP body, is the important part for a couple reasons:\n<\/p>\n\n
\n save money\n <\/li>\n
\n from the client’s internal cache. Because there’s no data transfer involved, this happens
\n lightning fast, making users happy and potentially saving some resources for them, too.\n<\/p>\nLast-Modified<\/code> and If-Modified-Since<\/code><\/h2>\nETag<\/code>, Google’s crawlers support Last-Modified based<\/code>
\n conditional requests, too, exactly as defined in the HTTP Caching standard. This works the same
\n way as ETag<\/code> from a semantic perspective — an identifier is used to decide
\n whether the resource is cacheable —, and provides the same benefits as ETag<\/code> on
\n the clients’ side.\n<\/p>\nLast-Modified<\/code> as a caching
\n directive:\n<\/p>\n\n
Last-Modified<\/code> header must be formatted according to the
\n HTTP standard<\/a>.
\n To avoid parsing issues, we recommend using the following date format:
\n “Weekday, DD Mon YYYY HH:MM:SS<\/span> Timezone”. For example,
\n “Fri, 4 Sep 1998 19:15:56 GMT<\/span>“.\n <\/li>\nmax-age<\/code> field of the
\n Cache-Control<\/code> header to help crawlers determine when to recrawl the specific URL.
\n Set the value of the max-age<\/code> field to the expected number of seconds the content
\n will be unchanged. For example, Cache-Control: max-age=94043<\/code>.\n <\/li>\n<\/ol>\nExamples<\/h2>\n
\n showing an example of the chain of requests and responses seems to help me. Here are two chains
\n — one for ETag<\/code>\/If-None-Match<\/code> and one for
\n Last-Modified<\/code>\/If-Modified-Since<\/code> — to visualize how it’s supposed
\n to work:\n<\/p>\n\n
\n <\/th>\n ETag<\/code>\/If-None-Match<\/code><\/th>\nLast-Modified<\/code>\/If-Modified-Since<\/code><\/th>\n<\/tr>\n\n \n A server’s response to a crawl:<\/strong> This is the response from which a crawler can
\n save the precondition header fields ETag<\/code> and Last-Modified<\/code>.\n <\/td>\n\n \n \n \n Subsequent crawler conditional request:<\/strong> The conditional request is based on
\n the precondition header values saved from a previous request. The values are sent back to the
\n server for validation in the If-None-Match<\/code> and If-Modified-Since<\/code>
\n request headers.\n <\/td>\n\n \n \n \n Server response to the conditional request:<\/strong> Since precondition header values
\n sent by the crawler are validated on the server’s side, the server returns a 304<\/code>
\n HTTP status code (without an HTTP body) to the crawler. This will happen to every subsequent
\n request until the preconditions fail to validate (the ETag<\/code> or the
\n Last-Modified<\/code> date changes on the server’s side).\n <\/td>\n\n \n
\n few bucks on your hosting bill, talk to your hosting or CMS provider, or your developers about how
\n to enable HTTP caching for your site. If nothing else, your users will like you a bit more.\n<\/p>\n
\n Search Central help community<\/a>, and if
\n you have comments about how we’re caching, leave feedback on
\n the documentation about caching<\/a>
\n that we published together with this blog post.\n<\/p>\n