Cloudflare enhances AI bots’ ability to block website scraping

Cloudflare, a global internet security company that claims to protect around 20% of the world’s web traffic, has launched what it calls an “easy button” for website owners who want to block AI services from accessing their content. The move comes as demand for content used to train AI models has skyrocketed.

Cloudflare’s core service, which acts as an internet proxy, scans and filters web traffic before it reaches websites. The company says that on average, its network handles more than 57 million requests per second.

“To keep the Internet safe for content creators, we’re launching an all-new ‘easy button’ that blocks all AI bots,” Cloudflare said in a statement Wednesday. “We know our customers don’t want AI bots visiting their websites, especially if they’re doing it fraudulently.”

While some AI companies do a good job of identifying web scraping bots and following website guidelines to avoid using them, not all are transparent about their activities.

The new simple setup is available to all Cloudflare customers, including our free tier.

AI bot activity analysis

Cloudflare shared a lot of information about AI crawler activity it observed across its systems along with the announcement.

According to Cloudflare data, AI bots accessed approximately 39% of the top 1 million “Internet properties” using Cloudflare in June. However, only 2.98% of those properties took action to block or dispute those requests. Cloudflare also notes that “the higher the ranking of an Internet property (the more popular it is), the more likely it is to be targeted by AI bots.”

The company said the most active crawlers are those operated by TikTok owner ByteDance, Amazon, Anthropic, and OpenAI. The most active crawler is ByteDance’s Bytespider, which tops the charts for number of requests, activity range, and block frequency. GPTBot, managed by OpenAI and used to gather training data for products like ChatGPT, came in second for both crawling activity and blocking.

Perplexity’s web crawler, which has recently been the subject of controversy for its content crawling practices, has been detected visiting only a small fraction of sites protected by Cloudflare.

Website owners can implement their own rules to block known web crawlers, but Cloudflare says most customers who do so only block popular AI developers like OpenAI, Google, and Meta, and not major crawlers from Bytedance or other companies.

AI vs AI

Cloudflare’s report highlighted the use of deceptive tactics by some AI bot operators to evade blocking measures, attempting to disguise crawler activity as legitimate web traffic.

“Unfortunately, we have observed bot operators attempting to use fake user agents to make their bots appear to be real browsers,” Cloudflare wrote.

As a result, AI is a key tool in the arsenal of companies that want to stop automated activity, whether it’s from AI developers, search engines, or malicious attackers. Cloudflare says it uses machine learning models to assign a “bot score” to each request to websites protected by its service, with a lower score indicating the activity is less likely to be legitimate.

Drawing on Cloudflare’s massive global Internet traffic data set, the model considers a number of signals, including the request’s IP address, user agent, and behavioral patterns, to determine a bot score.

To illustrate this, Cloudflare said it looked at traffic from a specific bot known for evasive behavior. The results were telling. All of the detections were rated below 30 out of 100, and most fell into the bottom two bands, scoring below 9. In other words, despite attempts to hide their origins, the bot’s activity patterns were revealing and Cloudflare was able to block it.

Web content protection

Generative AI models rely on a vast amount of existing content, much of it collected from across the web. To keep the AI ​​up to date, developers must continue to collect information at scale.

Website owners and content creators are pushing back, and big publishers like news organizations are taking legal action against AI companies. In the Perplexity case mentioned earlier, the following publishers: Forbes and mad It claims that the content is being taken and republished without permission. Music publisher Sony preemptively warned more than 700 tech companies to stay away in May, and Warner Music Group followed suit this week.

As AI increasingly provides information to users without citing its sources, it could pose an existential threat to publishers. A recent study by Rand Fishkin, CEO of SparkToro, found that 60% of people searching for information on Google stopped visiting the website that provided that information because Google’s AI immediately provided a summarized answer.

Edited by Ryan Ozawa.