Cloudflare’s AI Crawl Control update isn’t just a new set of options publishers can turn on if they get around to it — new default settings are rolling out that apply automatically to many sites on the platform, whether or not the site owner has actively configured anything. Our broader look at Cloudflare’s crawler categories covers the three-way classification system itself in more depth; this piece is specifically about what changes under the new defaults and how to get ready before they apply to your own site rather than after.
Why a Default Rollout Changes the Calculus
Optional settings tend to get ignored — most site owners simply never open the dashboard to configure something that isn’t currently broken. A default that shifts automatically removes that inertia entirely: once the new rules apply, some AI crawlers that could previously reach your content freely may find themselves blocked by default, and others that were blocked may find new categories to move through. Either direction is a real change to your traffic and visibility, and it happens whether you’ve reviewed your settings or not.
What Cloudflare’s New Approach Actually Classifies
Rather than treating every AI bot identically, Cloudflare separates crawler traffic into three categories: search crawlers that index content for traditional search engines, agent crawlers that retrieve information in real time for AI assistants and AI-powered applications, and training crawlers that collect content specifically to train large language models. Splitting these apart is what makes granular rules possible in the first place — a single allow/block toggle can’t distinguish “help my site get discovered” from “use my content to train a model I get no attribution or traffic from.”
What Changes Under the New Defaults
Under the new default settings, search crawlers continue accessing public pages as before — this part shouldn’t disrupt your normal search visibility. The more significant shifts:
AI training crawlers may be blocked by default on pages displaying advertisements. If your revenue model depends on ad-supported content, this default works in your favor without you having to configure anything — but it’s worth verifying it actually applies to your specific page templates rather than assuming.
AI agent crawlers can also be restricted, depending on preferences you’ve set — or haven’t set, in which case the platform default applies instead of your own judgment about the tradeoff.
Mixed-use crawlers — bots that combine search, agent, and training functions in a single crawler, which is increasingly common as AI companies consolidate their infrastructure — may also be blocked by default on ad-supported pages unless you’ve explicitly chosen different settings for that category.
The practical takeaway: if you haven’t touched your Cloudflare AI Crawl Control settings, the platform is making (or will soon make) several consequential decisions on your behalf. They may well be the decisions you’d have made anyway — but “probably fine” isn’t the same as verified, especially when the setting directly affects revenue.
What Counts as an “Ad-Supported Page”?
This distinction matters more than it might first appear, since several of the new defaults hinge specifically on it. A page is generally treated as ad-supported if it serves advertising units, and the classification typically applies at the page level rather than site-wide — meaning a blog with a mix of ad-supported posts and ad-free evergreen guides may see different crawler behavior across different sections of the same site. If your ad implementation is inconsistent across templates, it’s worth checking how Cloudflare’s system actually classifies each page type before relying on the defaults rather than assuming uniform treatment.
Preparing Your Site for the New Defaults
Audit your current AI Crawl Control settings. If you’ve never opened this panel, you’re currently on whatever default applied when your account was created — not necessarily the one that’s rolling out now.
Decide deliberately, not by default, for each crawler category. Search crawlers are usually a clear “allow” for most publishers. Training crawlers and agent crawlers deserve an actual decision based on your business model — ad-supported content generally benefits from restricting training crawlers, while agent crawlers might be worth allowing if you want your content surfaced (with attribution) in AI-powered search results. Our look at whether AI-generated content can rank on Google covers the visibility side of that same trade-off.
Check your ad-supported page classification. Confirm the pages you expect to be treated as ad-supported are actually configured that way in Cloudflare’s system, especially if you run a mix of templates across your site.
Review the enhanced analytics before and after any change. Cloudflare’s expanded reporting shows which AI companies visit your site, how often, whether they generate any referral traffic back to you, and which crawlers consume the most resources. Pulling a baseline before the new defaults apply makes it much easier to see what actually changed, rather than guessing after the fact.
Set a recurring reminder to revisit this periodically. Crawler classifications and AI company behavior are both moving targets — a setting that makes sense today may need revisiting down the line as new AI services emerge and existing ones change how they identify themselves.
What the Enhanced Analytics Actually Show
Beyond the rule changes themselves, Cloudflare is expanding what publishers can actually see: which specific AI companies are visiting, how frequently each one crawls, whether that traffic converts into any referral visits back to the site, and which crawlers are consuming the most server resources relative to the value they provide. This visibility is arguably as important as the rules themselves — you can’t make a genuinely informed allow/block decision about a crawler you didn’t know was visiting your site in the first place.
Frequently Asked Questions
Do I need to do anything, or will Cloudflare handle it automatically? The new defaults apply automatically, but “automatic” isn’t the same as “optimal for your specific site.” Reviewing your settings lets you make an intentional choice instead of inheriting whatever the platform-wide default happens to be.
Will this affect my normal search engine rankings? Traditional search crawler access isn’t changing under these defaults — the update specifically targets AI agent and training crawlers, not standard search indexing.
What happens if I do nothing at all? Your site will follow whatever default Cloudflare applies to accounts that haven’t configured custom settings — likely restricting training crawlers on ad-supported pages, per the general pattern described above, but worth confirming for your specific setup rather than assuming.
Is this only relevant to large publishers with a dedicated technical team? No — if your site runs on Cloudflare and carries any advertising, this affects you regardless of size. Independent bloggers arguably have more to gain from actively configuring these settings than large publishers with existing licensing deals already in place.
How is this different from Cloudflare’s general AI crawler framework? Our companion article covers the three-category classification system itself and the broader best-practice strategy around it. This piece focuses specifically on the new default rules and what to check before they apply to your site.
Final Thoughts
A default rollout changes an optional feature into something that deserves an actual review, not just awareness that it exists. Cloudflare’s new defaults will likely land reasonably well for most publishers out of the box — but “likely fine” and “verified” aren’t the same thing when the setting directly touches how AI companies access content you spent real time creating. A few minutes reviewing your AI Crawl Control settings is a small investment against a change that’s rolling out either way.
Related Articles