\n website and in general keeping web traffic-related headaches away. This is their primary purpose
\n after all: speedy delivery of your content even if your site is getting loads of traffic. The “D”<\/span>
\n in CDN is for delivering or distributing the content across the world, so transfer times to your
\n users is also lower than just hosting in one data center somewhere. In this post we’re going to
\n explore how to make use of CDNs in a way that improves crawling and users’ experience on your
\n site, and we also look at some nuances of crawling CDN-backed sites.\n<\/p>\n
Recap: What is a CDN?<\/h2>\n
\n CDNs are basically an intermediary between your origin server (where your website lives) and the \n CDNs can drastically speed up your site<\/strong> by serving users from a location that’s \n And finally, \n You might have the fastest servers and the best uplink money can buy and you might not think you \n In short, CDNs are your friend and if your site is large or you’re expecting (or even already \n On the crawling front, CDNs can also be helpful, but they can cause some crawling issues (albeit \n Our crawling infrastructure is designed to allow higher crawl rates on sites that are backed by a \n Say, you start a stock photo site today and happen to have 1,000,007 pictures in… stock. You \n However, on the first access of a URL the CDN’s cache is “cold”, meaning that since no one has \n In short, even if your webshop is backed by a CDN, your server will need to serve those 1,000,007 \n As we explained in our first \n If you back your main host with a CDN, then you avoid this problem: one hostname to query, and the \n In the end, choose the solution that works best for your business: have a separate hostname \n Due to the CDNs’ flood protection and how crawlers, well, crawl, occasionally the bots that you do \n Hard blocks are when the CDN sends a response to a crawl request that’s an error in some form. \n A similar issue may pop up (pun very much intended) when your CDN shows those “are you sure you’re \n Our crawlers are in fact convinced that they’re NOT human and they’re not pretending to be one. \n In case of both hard and soft blockages, the easiest way to check if things are working correctly \n Additionally, to help with these unintended blockages, Google, other search engines, and other \n If you need your site to show up in search engines, we strongly recommend checking whether the \n This was the last post in our
\n end user, and serves (some) files for them. Historically,
\n CDNs’ biggest focus is caching<\/strong>, meaning that once a user requested a URL from
\n your site, CDNs will store the contents of that URL in their caches for a time so your server
\n doesn’t have to serve that file again for a while.\n<\/p>\n
\n close to them. Say, if a user in Australia is accessing a site hosted in Germany, a CDN will serve
\n that user from their caches in Australia, cutting down the roundtrip across the globe. Lightspeed
\n or not,
\n the distance is still quite large<\/a>.\n<\/p>\n
\n CDNs are a fantastic tool to protect your site from being overloaded and some security threats<\/strong>.
\n With the amount of global traffic CDNs manage, they can construct reliable traffic models to
\n detect traffic anomalies and block accesses that seem excessive or malicious. For example, on
\n October 21, 2024,
\n Cloudflare’s systems<\/a>
\n autonomously detected and mitigated a 4.2
\n Tbps<\/a>
\n (ed: that’s a lot)<\/em> DDoS attack that lasted around a minute.\n<\/p>\nHow CDNs can help your site<\/h2>\n
\n need to speed up anything, but CDNs can save you money in the long run, especially if your site is
\n big:\n<\/p>\n\n
\n HTML are served from a CDN’s caches, your servers don’t have to spend compute and bandwidth on
\n serving those resources, reducing server load in the process. This usually also means that pages
\n load faster in users’ browsers, which
\n correlates with better conversions<\/a>.\n <\/li>\n
\n blocking excessive or malicious traffic, letting your users visit your site even when
\n misbehaving bots or no-good-doers would overload your servers.
\n Besides flood protection, the same controls that are used to block bad traffic can also be used
\n for blocking traffic that you simply don’t want, be that certain crawlers, clients that fit in a
\n certain pattern, or just trolls that keep using the same IP address. While you can do this on
\n your server or firewall too, it’s usually much easier to use a CDN’s user interface.\n <\/li>\n
\n This of course might only work for static content, but that might already be enough to ensure
\n they don’t take their business somewhere else.\n <\/li>\n<\/ul>\n
\n receiving!) large amounts of traffic, you might want to find one that fits your needs based on
\n factors such as price, performance, reliability, security, customer support, scalability, future
\n expansion. Check with your hosting or CMS provider, to learn your options (and whether you already
\n use one).\n<\/p>\nHow crawling affects sites with CDNs<\/h2>\n
\n rarely). Stay with us.\n<\/p>\nCDNs’ effect on crawl rate<\/h3>\n
\n CDN, which is inferred from the IP address of the service that’s serving the URLs our crawlers are
\n accessing. This works well, at least most of the time.\n<\/p>\n
\n launch your website with a landing page, category pages, and detail pages for all of your stuff
\n — so you end up with a lot of pages. We explain in our documentation on
\n crawl capacity limit<\/a>
\n that while Google Search would like to crawl all of these pages as quickly as possible, crawling
\n should also not overwhelm your servers. If your server starts responding slowly when facing an
\n increased number of crawling requests, throttling is applied on Google’s side to prevent your
\n server from getting overloaded. The threshold for this throttling is much higher when our crawling
\n infrastructure detects that your site is backed by a CDN, and assumes that it’s fine to send more
\n simultaneous requests because your server most likely can handle it, thus crawling your webshop
\n faster.\n<\/p>\n
\n requested that URL yet, its contents weren’t cached by the CDN yet, so your origin server will
\n still need serve that URL at least once to “warm up” the CDN’s cache. This is very similar to
\n how HTTP caching works<\/a>, too.\n<\/p>\n
\n URLs at least once. Only after that initial serve can your CDN help you with its caches. That’s a
\n significant burden on your “crawl budget” and the crawl rate will likely be high for a few days;
\n keep that in mind if you’re planning to launch many URLs at once.\n<\/p>\nCDNs’ effect on rendering<\/h3>\n
\n Crawling December blog post about resource crawling<\/a>,
\n splitting out resources to their own hostname or a CDN hostname (cdn.example.com<\/code>) may
\n allow our Web Rendering Service (WRS) to render your pages more efficiently. This comes with a
\n caveat though: this practice may negatively affect page performance due to the overhead of a
\n connection to a different hostname, so you need to carefully consider
\n page experience<\/a> with rendering performance.\n<\/p>\n
\n critical rendering resources are likely served from the CDN’s cache so your server doesn’t need to
\n serve them (and no hit on page experience).\n<\/p>\n
\n (cdn.example.com<\/code>) for static resources, back your main hostname with a CDN, or do
\n both. Google’s crawling infrastructure supports either option without issues.\n<\/p>\nWhen CDNs are overprotective <\/h2>\n
\n want on your site may end up in your CDN’s blocklist, typically in their Web Application Firewall
\n (WAF). This prevents crawlers from accessing your site, which ultimately may prevent your site
\n from showing up in search results. The block can happen in various ways, some more harmful for a
\n site’s presence in Google’s search results than others, and it can be tricky (or impossible) for
\n you to control since they happen on the CDN’s end. For the purpose of this blog post we put them
\n in two buckets: hard blocks and soft blocks.\n<\/p>\nHard blocks<\/h3>\n
\n These can be:\n<\/p>\n\n
503<\/code>\/429<\/code> status codes<\/strong>: Sending
\n these status codes is the preferred way<\/a>
\n to signal a temporary blockage. It will give you some time to react to unintended blocks by the
\n CDN.\n <\/li>\n
\n be removed from Google’s search index, as these
\n network errors are considered terminal, “hard” errors<\/a>.
\n Additionally they may also considerably affect your site’s crawl rate because they signal our
\n crawl infrastructure that the site is overloaded.\n <\/li>\n200<\/code> status code<\/strong>: Also known as
\n soft errors<\/a>,
\n this is particularly bad. If the error message is equated on Google’s end to a “hard” error
\n (say, an HTTP 500<\/code>), Google will remove the URL from Search. If Google couldn’t
\n detect the error messages as “hard” errors, all the pages with the same error message may be
\n eliminated as duplicates from Google’s search index. Since Google indexing has little incentive
\n to request a recrawl of duplicate URLs, recovering from this may take more time.\n <\/li>\n<\/ul>\nSoft blocks<\/h3>\n
\n a human” interstitials.\n<\/p>\n![\"Crawley](\"https:\/\/alienroad.com\/wp-content\/uploads\/kb-gorsel\/g-a4784825b66d.png\")
<\/p>\n
\n They just wanna crawl. However when the interstitial shows up, that’s all they see, not your
\n awesome site. In case of these bot-verification interstitials, we strongly recommend sending a
\n clear signal in the form of a 503 HTTP status code to automated clients like crawlers that the
\n content is temporarily unavailable. This will ensure that the content is not removed from Google’s
\n index automatically.\n<\/p>\nDebugging blockages<\/h3>\n
\n is to use the
\n URL Inspection tool in Search Console<\/a>
\n and observe the rendered image: if it shows your page, you’re good; if it shows an empty page, an
\n error, or a page with a bot challenge, you might want to talk to your CDN about it.\n<\/p>\n
\n crawler operators publish
\n our IP addresses<\/a> to help you to
\n identify our crawlers and, if you feel that’s appropriate, remove the blocked IPs from the WAF
\n rules, or even allowlist them. Where you can do this depends on the CDN you’re using; fortunately
\n most CDNs and standalone WAFs have fantastic documentation. Here’s some we could find with a
\n little searching (as of publication of this post):\n<\/p>\n\n
\n crawlers you care about can access your site. Remember that the IPs may end up on a blocklist
\n automatically, without you knowing, so checking in on the blocklists every now and then is a good
\n idea for your site’s success in search and beyond. If the blocklist is very long (not unlike this
\n blog post), try to look for just the first few segments of the IP ranges, for example, instead of
\n looking for 192.168.0.101<\/code> you can just look for 192.168<\/code>.\n<\/p>\n
\n Crawling December blog post series<\/a>,
\n we hope you enjoyed them as much as we loved writing them. If you have… blah blah blah… you
\n know the drill.\n<\/p>\n