Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • ADOPTION
  • TRADING
  • HACKING
  • SLOT
Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • ADOPTION
  • TRADING
  • HACKING
  • SLOT
Crypto Flexs
Home»ETHEREUM NEWS»Cloudflare enhances AI bots’ ability to block website scraping
ETHEREUM NEWS

Cloudflare enhances AI bots’ ability to block website scraping

By Crypto FlexsJuly 5, 20244 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr Email
Cloudflare enhances AI bots’ ability to block website scraping
Share
Facebook Twitter LinkedIn Pinterest Email

Cloudflare, a global internet security company that claims to protect around 20% of the world’s web traffic, has launched what it calls an “easy button” for website owners who want to block AI services from accessing their content. The move comes as demand for content used to train AI models has skyrocketed.

Cloudflare’s core service, which acts as an internet proxy, scans and filters web traffic before it reaches websites. The company says that on average, its network handles more than 57 million requests per second.

“To keep the Internet safe for content creators, we’re launching an all-new ‘easy button’ that blocks all AI bots,” Cloudflare said in a statement Wednesday. “We know our customers don’t want AI bots visiting their websites, especially if they’re doing it fraudulently.”

While some AI companies do a good job of identifying web scraping bots and following website guidelines to avoid using them, not all are transparent about their activities.

The new simple setup is available to all Cloudflare customers, including our free tier.

AI bot activity analysis

Cloudflare shared a lot of information about AI crawler activity it observed across its systems along with the announcement.

According to Cloudflare data, AI bots accessed approximately 39% of the top 1 million “Internet properties” using Cloudflare in June. However, only 2.98% of those properties took action to block or dispute those requests. Cloudflare also notes that “the higher the ranking of an Internet property (the more popular it is), the more likely it is to be targeted by AI bots.”

The company said the most active crawlers are those operated by TikTok owner ByteDance, Amazon, Anthropic, and OpenAI. The most active crawler is ByteDance’s Bytespider, which tops the charts for number of requests, activity range, and block frequency. GPTBot, managed by OpenAI and used to gather training data for products like ChatGPT, came in second for both crawling activity and blocking.

Image: Cloudflare

Perplexity’s web crawler, which has recently been the subject of controversy for its content crawling practices, has been detected visiting only a small fraction of sites protected by Cloudflare.

Image: Cloudflare

Website owners can implement their own rules to block known web crawlers, but Cloudflare says most customers who do so only block popular AI developers like OpenAI, Google, and Meta, and not major crawlers from Bytedance or other companies.

AI vs AI

Cloudflare’s report highlighted the use of deceptive tactics by some AI bot operators to evade blocking measures, attempting to disguise crawler activity as legitimate web traffic.

“Unfortunately, we have observed bot operators attempting to use fake user agents to make their bots appear to be real browsers,” Cloudflare wrote.

As a result, AI is a key tool in the arsenal of companies that want to stop automated activity, whether it’s from AI developers, search engines, or malicious attackers. Cloudflare says it uses machine learning models to assign a “bot score” to each request to websites protected by its service, with a lower score indicating the activity is less likely to be legitimate.

Drawing on Cloudflare’s massive global Internet traffic data set, the model considers a number of signals, including the request’s IP address, user agent, and behavioral patterns, to determine a bot score.

Image: Cloudflare

To illustrate this, Cloudflare said it looked at traffic from a specific bot known for evasive behavior. The results were telling. All of the detections were rated below 30 out of 100, and most fell into the bottom two bands, scoring below 9. In other words, despite attempts to hide their origins, the bot’s activity patterns were revealing and Cloudflare was able to block it.

Web content protection

Generative AI models rely on a vast amount of existing content, much of it collected from across the web. To keep the AI ​​up to date, developers must continue to collect information at scale.

Website owners and content creators are pushing back, and big publishers like news organizations are taking legal action against AI companies. In the Perplexity case mentioned earlier, the following publishers: Forbes and mad It claims that the content is being taken and republished without permission. Music publisher Sony preemptively warned more than 700 tech companies to stay away in May, and Warner Music Group followed suit this week.

As AI increasingly provides information to users without citing its sources, it could pose an existential threat to publishers. A recent study by Rand Fishkin, CEO of SparkToro, found that 60% of people searching for information on Google stopped visiting the website that provided that information because Google’s AI immediately provided a summarized answer.

Edited by Ryan Ozawa.

Generally intelligent Newsletter

A weekly AI journey explained by Gen, a generative AI model.

Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

Bitcoin dives for less than $ 102K in fading momentum and macroscopic uncertainty.

June 6, 2025

Enterprise Ethereum Alliance accelerates strategic vision with new leadership and board promises -Enterprise Ethereum Alliance

June 5, 2025

Checkpoint #3: June 2025 Stats Ether Leeum Foundation Blog

June 4, 2025
Add A Comment

Comments are closed.

Recent Posts

Defi, not MICA II at the front line

June 6, 2025

Improve video content with VEO 3: Comprehensive guide to effective prompt

June 6, 2025

Is the trump Fuel Fuel Dogecoin in ELON Musk less than $ 0.10?

June 6, 2025

The HBAR price aims for $ 0.40 when a big decision of the SEC comes.

June 6, 2025

Why is the ecosystem of the SEI network boom in the second quarter?

June 6, 2025

Why is the ecosystem of the SEI network boom in the second quarter?

June 6, 2025

Start BTC, DOGE and ETH mining today!

June 6, 2025

AI increases artistry as an innovative creation at NVIDIA GTC Paris.

June 6, 2025

JPMORGAN accepts Bitcoin ETF with loan collateral: How investors can benefit

June 6, 2025

NVIDIA’s BLACKWELL Architecture MLPERF has set up new performance standards.

June 6, 2025

ZK-Proof Blockchain Altcoin Lagrange (LA) New Coinbase

June 6, 2025

Crypto Flexs is a Professional Cryptocurrency News Platform. Here we will provide you only interesting content, which you will like very much. We’re dedicated to providing you the best of Cryptocurrency. We hope you enjoy our Cryptocurrency News as much as we enjoy offering them to you.

Contact Us : Partner(@)Cryptoflexs.com

Top Insights

Defi, not MICA II at the front line

June 6, 2025

Improve video content with VEO 3: Comprehensive guide to effective prompt

June 6, 2025

Is the trump Fuel Fuel Dogecoin in ELON Musk less than $ 0.10?

June 6, 2025
Most Popular

Mnemonic Seed – Python “pbkdf2_hmac” function does not return the same as Ian Coleman BIP39 tool. Why?

July 18, 2024

5 Best Cryptocurrencies with Huge Profit Potential in April

April 7, 2024

The analyst predicts the incoming escape of Floki, and Dogecoin and Chainlink prepares for the reversal.

February 16, 2025
  • Home
  • About Us
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms and Conditions
© 2025 Crypto Flexs

Type above and press Enter to search. Press Esc to cancel.