Cloudflare to block mixed-purpose bots from ad-supported sites by default

Cloudflare announced plans to stop mixed-use crawlers from reaching ad-supported customer websites without explicit approval, as part of its drive to hand publishers greater authority over AI interactions.

Cloudflare targets mixed-use crawlers starting September 15, 2026. Beginning on that date, new customers and newly added sites will automatically permit search indexing while denying access for training and agent activity on monetized pages. Free-tier users who left their configurations untouched will receive the updated defaults as well.

The provider says the policy guarantees that revenue-generating material stays shielded unless owners grant permission. Existing Cloudflare customers retain the ability to override the defaults and restore crawler access to those pages.

Apple, Google, and Microsoft crawlers could be affected. Crawlers run by Apple, Google, and Microsoft’s Bing risk being caught by the new stance, the company reported. All three companies provide an AI-specific opt-out mechanism that might spare them from enforcement.

Googlebot merges search-index duties with AI-training data collection. Many publishers have allowed the bot to continue because excluding it could remove their sites from Google Search. Microsoft’s Bingbot faces the same dynamic.

Applebot also handles both indexing and AI data gathering. Apple has expanded its Applebot crawler to collect material for AI systems alongside traditional indexing. The iBiz reported in June that "The data crawled by Applebot may also be used to help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools."

Publishers use robots.txt but many crawlers ignore it. Apple and Google honor robots.txt instructions that let owners block AI harvesting through Applebot-Extended and Google-Extended tokens. Bing respects a noarchive directive in the robots meta tag for the same purpose. Numerous other operators routinely disregard the voluntary standard, prompting Cloudflare to supply a firmer enforcement layer.

Matthew Prince, co-founder and CEO of Cloudflare, stated that because most internet traffic is now non-human the firm must move faster to foster a viable online environment. Cloudflare is also renaming its “Pay Per Crawl” feature to “Pay Per Use.” Prince added that the firm’s updated offerings and alliances deliver publishers better insight, fresh commercial avenues, and incentives for AI operators whose bots declare their purposes openly.

The company hopes the revised defaults will push mixed-purpose bots to disentangle search functions from training and agent roles.