Cloudflare Blocks AI Bots by Default: Is Your Site Invisible?

The shift in July 2025 marks a fundamental change in how content delivery networks handle automated traffic. For years, the internet operated on an open-crawling assumption where bots were allowed unless specifically forbidden. Cloudflare changed this for new domains to give site owners more control over how their intellectual property is used to train generative models.

AI companies often scrape massive amounts of data without providing direct referral traffic back to the source. By turning on these blocks by default, the network provider protects smaller publishers who might not have the technical expertise to configure complex firewall rules. This move also aligns with the reported Pay Per Crawl model, where AI agents may eventually need to compensate creators for high-volume data access.

Visibility in the modern era is no longer limited to the blue links on a Google results page. Many users now turn to AI chat interfaces to find products, services, and information. If Cloudflare is blocking these bots at the edge, your brand will not appear in the citations or answers provided by these tools.

For example, a Shopify store selling specialized outdoor gear might have excellent SEO, but if an AI bot cannot access the product descriptions, that store will be absent from AI-generated shopping recommendations. This creates a strategic dilemma for digital marketers who must balance data protection against the need for brand discovery in new search mediums.

The Pay Per Crawl concept is a reported vendor feature that introduces a marketplace for web data. Instead of allowing free and unlimited scraping, this model allows site owners to set terms for how AI companies access their content. It treats web data as a premium asset rather than a public utility.

This system aims to solve the value gap between content creators and AI developers. While still in its early stages, the integration of this model into the Cloudflare ecosystem suggests a future where high-quality data is gated behind commercial agreements. If your site is on a new domain, you are already part of this restricted ecosystem by default.

A common misconception is that allowing a bot in robots.txt is enough to ensure access. However, CDN and WAF blocks operate at a lower level of the networking stack. If Cloudflare identifies a request as coming from a known AI bot and the block is active, the request is terminated before it ever reads your robots.txt file.

Robots.txt is a set of instructions that polite bots choose to follow. In contrast, a Cloudflare block is an enforcement mechanism. If you want to be visible to AI, you must ensure that both your server-level instructions and your network-level firewall rules are in sync. Relying only on the robots.txt file is no longer a reliable strategy for modern web management.

To determine your current status, you need to access the security settings of your domain. Most users will find these settings under the ‘Bots’ section of their dashboard. Furthermore, you should look for specific logs that show challenged or blocked requests from known User-Agents associated with AI companies.

• Log into the Cloudflare dashboard and select your specific domain.
• Navigate to the Security menu on the left and click on the Bots sub-section.
• Check the status of the ‘AI Scrapers and Crawlers’ toggle.
• Review the Firewall Events log to see if ‘Verified Bot’ or ‘AI Bot’ categories are being blocked.
• Verify if any Custom WAF rules are explicitly targeting User-Agents like ‘GPTBot’ or ‘CCBot’.

Granular control is possible through custom firewall rules. You might want to allow ChatGPT to crawl your site for real-time search results but block other bots that only scrape for offline model training. This allows you to participate in the AI search economy without giving away your data for free to every crawler on the web.

In a typical account, you can create a Skip rule in the WAF. This rule can be configured to bypass the AI block if the User-Agent matches a specific string. However, be aware that some bots spoof their identity, so relying on User-Agents alone can be risky. Cloudflare’s verified bot list is a safer way to manage these permissions.

The decision to remain invisible to AI depends on your business model. If you run a news site or a blog that relies on ad revenue, AI scraping can be detrimental because it provides the information to the user without the user ever visiting your site. In this case, the default block is a significant advantage.

On the other hand, if you are a service provider or a local business, you likely want AI bots to know about your existence. If someone asks an AI assistant for the best digital marketing agency in a specific region, you want your data to be available to that model. Being invisible in that context means losing potential leads to competitors who have opted into AI crawling.

Checklist for AI visibility management

• Determine if your domain was registered after July 2025 to see if default blocks apply.
• Audit your traffic logs to see if AI crawlers are currently hitting a 403 Forbidden error.
• Evaluate the referral traffic potential from AI search engines like Perplexity.
• Set up a custom WAF rule to allow ‘Verified Bots’ if you want controlled visibility.
• Monitor the ‘Pay Per Crawl’ settings for future monetization opportunities.