Cloudflare launches 'bot blocker': What it means for AI companies like Google, OpenAI
Cloudflare , a major internet company, is set to fundamentally change how artificial intelligence (AI) companies access web content, potentially shaking up how AI models are trained. Cloudflare will block AI crawlers by default from accessing content without a website owner's explicit permission or compensation.
What are AI Crawlers and how Cloudflare’s new tool works
AI crawlers are automated bots that extract vast amounts of data from websites and other sources to train large language models (LLMs) used by companies like OpenAI and Google . This new tool means every new website signing up with Cloudflare will be prompted to decide if they want to allow AI crawlers. This move effectively gives website owners the power to stop bots from scraping their data.
Cloudflare, known as a content delivery network (CDN), plays a crucial role in delivering online content quickly and efficiently. Given that roughly 16% of global internet traffic flows through Cloudflare's CDN, this change carries significant weight.
What is changing
Cloudflare is also introducing a "pay per crawl" model, allowing publishers to charge AI crawlers for access to their content. Traditionally, the internet rewarded content creators by directing users to their original websites.
However, Cloudflare argues that AI crawlers are disrupting this model by collecting text, articles, and images to generate responses that users don't need to visit the original source for. This practice, according to Cloudflare, deprives publishers of crucial traffic and, consequently, advertising revenue.
“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate,” stated Matthew Prince, Cloudflare's co-founder and CEO. He added that this is about "safeguarding the future of a free and vibrant Internet with a new model that works for everyone."
What are AI Crawlers and how Cloudflare’s new tool works
Cloudflare, known as a content delivery network (CDN), plays a crucial role in delivering online content quickly and efficiently. Given that roughly 16% of global internet traffic flows through Cloudflare's CDN, this change carries significant weight.
What is changing
Cloudflare is also introducing a "pay per crawl" model, allowing publishers to charge AI crawlers for access to their content. Traditionally, the internet rewarded content creators by directing users to their original websites.
However, Cloudflare argues that AI crawlers are disrupting this model by collecting text, articles, and images to generate responses that users don't need to visit the original source for. This practice, according to Cloudflare, deprives publishers of crucial traffic and, consequently, advertising revenue.
“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate,” stated Matthew Prince, Cloudflare's co-founder and CEO. He added that this is about "safeguarding the future of a free and vibrant Internet with a new model that works for everyone."
Next Story