Cloudflare disables free AI access to content.

For over 25 years, the internet was defined by a secret bargain in which content providers permitted businesses such as Google to scan their websites in exchange for traffic. This traffic drove advertising, subscriptions, and online growth. However, with the emergence of artificial intelligence, this arrangement has come apart.
On July 1, Cloudflare, one of the internet’s primary infrastructure providers, declared that AI crawlers would be barred from accessing websites on its network unless they paid content authors. CEO Matthew Prince declared it “Content Independence Day.”

AI solutions such as ChatGPT and Google’s AI Overviews are progressively bypassing traditional search by immediately answering user inquiries, drawing on scraped web material and delivering little or no traffic to the source. According to Cloudflare, Google produces 750 times more traffic than OpenAI, while Anthropic is even lower, 30,000 times less.

“Instead of a fair trade, the web is being strip-mined by AI crawlers,” Prince said on his website. Content producers supply fuel for AI systems but are excluded from the ensuing value chain.

A New Business Model for Content Access.
Cloudflare controls access to around 20% of all websites worldwide. Using this size, the startup intends to launch a marketplace where AI companies may directly license material from authors. The methodology would reward material not for virality, but for its usefulness in filling AI knowledge gaps—a system analogously described as “filling holes in a block of Swiss cheese.”

This is the first time an internet infrastructure business has modified its default behavior to prevent AI access. While OpenAI and other companies have struck paid content arrangements with sites such as Reddit and the Financial Times, Cloudflare’s change is more comprehensive and systemic.
According to Cloudflare, this paradigm might usher in a “new golden age of content creation,” in which quality and depth are more important than traffic volume. However, this vision creates challenging questions: Who defines “valuable content”? How is compensation determined? Could this fracture the internet even more?