Cloudflare gives AI crawlers a September deadline: pay publishers or get blocked

Cloudflare has set the AI industry a deadline. From September, it will block the crawlers that hoover up content for AI training. Any page that carries ads becomes off-limits, unless the site’s owner says otherwise. The pitch is simple: stop giving the web away for free.

The company sits in front of a large share of the world’s web traffic. It announced the change on Wednesday. From 15 September, new Cloudflare sites will keep letting search engines index their pages. They will block AI training and AI agents from any page with advertising by default.

The rule also catches “mixed-use” crawlers, the bots that blend search, training, and agent tasks into one. If a crawler will not let a site owner separate those uses, it gets blocked on ad-supported pages.

The defaults apply to new customers and to new sites from existing customers. They also cover every free user who has not changed their settings. Owners can always let the bots back in from their dashboard. But the starting position has flipped. Content that earns money is now off-limits to AI unless its owner opts in.

Why now

Cloudflare’s argument rests on a stark number. Automated bots now drive more than half of all web traffic, a milestone the company says arrived earlier than expected. Chief executive Matthew Prince said most internet traffic is now non-human. Cloudflare, he argued, “must go further and act faster so that a sustainable ecosystem can emerge.”

The deeper problem is a trap publishers know well. Most sites want to appear in AI answers, just as they want to rank in search. But the same crawl often feeds a model that then answers the user directly. The visit, and the ad revenue, never arrive.

Cloudflare singled out the “world’s largest search engine,” a clear jab at Google. Its Googlebot blends indexing with AI training. That gives Google roughly twice the data access of rival AI firms. Blocking the bot risks vanishing from search. Microsoft’s Bing and Apple’s Applebot raise the same dilemma.

From tollbooth to meter

Blocking is only half the plan. Cloudflare is turning last year’s “Pay Per Crawl” tollbooth into “Pay Per Use.” It now pays publishers when their content shapes an AI answer, not just when it is fetched. Early partners are the AI search firms Ceramic.ai and You.com.

Cloudflare is also adding a dashboard so publishers can see which bots take their work and how little traffic those firms send back. It gives the behaviour a name, Answer Engine Optimisation, the AI-era heir to SEO.

That reframing lands in a market already tilting this way. A wave of startups now sells tools to help brands stay visible inside chatbots, betting that GEO is the new SEO. Cloudflare wants to own the plumbing beneath it.

The open web at stake

The backdrop is grim for publishers. AI-generated answers are cutting the clicks that fund the web. They keep users on Google or inside a chatbot, rather than on the sites that did the work. One field study found Google’s AI Overviews cut outbound clicks by about 40 per cent. Economists have even started to model an outright collapse of the open web if the bargain is not repaired.

Whether one company can repair it is doubtful. Google and Apple already offer opt-out crawlers that may slip past Cloudflare’s block, and rivals could route around it. Regulators are circling the same problem from another angle. The UK is forcing Google to let publishers opt out of AI search without losing their ranking, and news publishers are suing OpenAI over training.

Cloudflare’s move is the most aggressive attempt yet to make AI pay for what it reads. The deadline is 15 September. The rest of the web will be watching what the AI giants do next.