Cloudflare adds granular AI traffic controls for all users

Cloudflare has expanded its traffic management options to give website owners more control over how AI systems interact with their content. The changes, announced as part of the company’s second annual Content Independence Day initiative, replace a previous all-or-nothing blocking approach with category-specific settings for search, agent, and training bots. Customers can now apply distinct rules to each type of AI traffic while also protecting pages that rely on advertising revenue.

What changed

The new controls are available to all Cloudflare customers without additional fees. Users can access the settings through the dashboard or API, where AI traffic is now broken into three subcategories: search crawlers (e.g., Googlebot), AI agents (e.g., chatbots or research assistants), and training bots (e.g., systems scraping content for model training). A fourth option allows site owners to shield ad-monetized pages from AI traffic that might bypass revenue-generating interactions.

Cloudflare’s previous system offered a single toggle to block all AI bots, which some customers found too blunt for their needs. The updated interface retains the ability to block all AI traffic but adds granularity for those who want to permit certain types of interactions while restricting others. For example, a publisher might allow search crawlers to index content while blocking training bots to prevent unauthorized use in AI datasets.

Why the update matters

The shift reflects growing tension between content creators and AI developers over how web data is used. Many publishers and independent site owners have expressed concerns about AI systems scraping their work without permission or compensation, particularly when the data is repurposed for commercial AI training. At the same time, some organizations want to maintain visibility in search results or allow specific AI tools to access their content for legitimate purposes, such as research or customer service.

Cloudflare’s move to categorize AI traffic more precisely gives site owners a middle ground. Instead of choosing between full access or complete blocking, they can now tailor their policies to their business models. For instance, a news site might block training bots to protect its journalism while still allowing search engines to crawl its articles. Similarly, an e-commerce platform could permit AI agents to answer customer queries about products while restricting training bots from harvesting product descriptions for competitive use.

The update also addresses a practical concern for ad-supported sites. AI bots that scrape content without loading ads can deprive publishers of revenue, even if the traffic appears legitimate. By adding protections for ad-monetized pages, Cloudflare enables site owners to mitigate this issue without blocking all AI traffic outright.

What to watch

The new controls could influence how other content delivery networks and web infrastructure providers approach AI traffic management. If Cloudflare’s granular settings gain traction, competitors may follow suit with similar categorization systems. This could lead to a broader shift in how AI interactions with web content are governed, with potential implications for both AI developers and content creators.

For now, the feature is available to all Cloudflare customers, including those on free plans. The company has not announced plans to charge for the functionality, but future pricing changes remain possible as the AI landscape evolves. Site owners using Cloudflare should review their traffic settings to determine whether the new options align with their goals for content distribution and monetization.