Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • ADOPTION
  • TRADING
  • HACKING
  • SLOT
  • TRADE
Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • ADOPTION
  • TRADING
  • HACKING
  • SLOT
  • TRADE
Crypto Flexs
Home»ETHEREUM NEWS»Cloudflare enhances AI bots’ ability to block website scraping
ETHEREUM NEWS

Cloudflare enhances AI bots’ ability to block website scraping

By Crypto FlexsJuly 5, 20244 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr Email
Cloudflare enhances AI bots’ ability to block website scraping
Share
Facebook Twitter LinkedIn Pinterest Email

Cloudflare, a global internet security company that claims to protect around 20% of the world’s web traffic, has launched what it calls an “easy button” for website owners who want to block AI services from accessing their content. The move comes as demand for content used to train AI models has skyrocketed.

Cloudflare’s core service, which acts as an internet proxy, scans and filters web traffic before it reaches websites. The company says that on average, its network handles more than 57 million requests per second.

“To keep the Internet safe for content creators, we’re launching an all-new ‘easy button’ that blocks all AI bots,” Cloudflare said in a statement Wednesday. “We know our customers don’t want AI bots visiting their websites, especially if they’re doing it fraudulently.”

While some AI companies do a good job of identifying web scraping bots and following website guidelines to avoid using them, not all are transparent about their activities.

The new simple setup is available to all Cloudflare customers, including our free tier.

AI bot activity analysis

Cloudflare shared a lot of information about AI crawler activity it observed across its systems along with the announcement.

According to Cloudflare data, AI bots accessed approximately 39% of the top 1 million “Internet properties” using Cloudflare in June. However, only 2.98% of those properties took action to block or dispute those requests. Cloudflare also notes that “the higher the ranking of an Internet property (the more popular it is), the more likely it is to be targeted by AI bots.”

The company said the most active crawlers are those operated by TikTok owner ByteDance, Amazon, Anthropic, and OpenAI. The most active crawler is ByteDance’s Bytespider, which tops the charts for number of requests, activity range, and block frequency. GPTBot, managed by OpenAI and used to gather training data for products like ChatGPT, came in second for both crawling activity and blocking.

Image: Cloudflare

Perplexity’s web crawler, which has recently been the subject of controversy for its content crawling practices, has been detected visiting only a small fraction of sites protected by Cloudflare.

Image: Cloudflare

Website owners can implement their own rules to block known web crawlers, but Cloudflare says most customers who do so only block popular AI developers like OpenAI, Google, and Meta, and not major crawlers from Bytedance or other companies.

AI vs AI

Cloudflare’s report highlighted the use of deceptive tactics by some AI bot operators to evade blocking measures, attempting to disguise crawler activity as legitimate web traffic.

“Unfortunately, we have observed bot operators attempting to use fake user agents to make their bots appear to be real browsers,” Cloudflare wrote.

As a result, AI is a key tool in the arsenal of companies that want to stop automated activity, whether it’s from AI developers, search engines, or malicious attackers. Cloudflare says it uses machine learning models to assign a “bot score” to each request to websites protected by its service, with a lower score indicating the activity is less likely to be legitimate.

Drawing on Cloudflare’s massive global Internet traffic data set, the model considers a number of signals, including the request’s IP address, user agent, and behavioral patterns, to determine a bot score.

Image: Cloudflare

To illustrate this, Cloudflare said it looked at traffic from a specific bot known for evasive behavior. The results were telling. All of the detections were rated below 30 out of 100, and most fell into the bottom two bands, scoring below 9. In other words, despite attempts to hide their origins, the bot’s activity patterns were revealing and Cloudflare was able to block it.

Web content protection

Generative AI models rely on a vast amount of existing content, much of it collected from across the web. To keep the AI ​​up to date, developers must continue to collect information at scale.

Website owners and content creators are pushing back, and big publishers like news organizations are taking legal action against AI companies. In the Perplexity case mentioned earlier, the following publishers: Forbes and mad It claims that the content is being taken and republished without permission. Music publisher Sony preemptively warned more than 700 tech companies to stay away in May, and Warner Music Group followed suit this week.

As AI increasingly provides information to users without citing its sources, it could pose an existential threat to publishers. A recent study by Rand Fishkin, CEO of SparkToro, found that 60% of people searching for information on Google stopped visiting the website that provided that information because Google’s AI immediately provided a summarized answer.

Edited by Ryan Ozawa.

Generally intelligent Newsletter

A weekly AI journey explained by Gen, a generative AI model.

Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

Checkpoint #4: Berlinterop | Ether Leeum Foundation Blog

June 28, 2025

Crypto Scam Markets thrives again after attempting to clean up Telegram.

June 24, 2025

Bitcoin is more than $ 104K as the merchant’s eyes move from H2 to $ 145,000.

June 20, 2025
Add A Comment

Comments are closed.

Recent Posts

Elon Musk, SpaceX And Crypto Hype: What’s Really Going On?

June 28, 2025

Checkpoint #4: Berlinterop | Ether Leeum Foundation Blog

June 28, 2025

TRON Price Propects USDT supply exceeded $ 80 billion

June 28, 2025

Stablecoin startups surpass 2021 venture capital peaks as institutional money spills.

June 28, 2025

No Altcoin Season 2025 ? Why Bitcoin Dominance Is Holding Strong In The Crypto Market

June 28, 2025

Why It Matters For Every Crypto Investor

June 27, 2025

Why It Matters For Every Crypto Investor

June 27, 2025

Safe smart account audit summary

June 27, 2025

CARV’s New Roadmap Signals Next Wave Of Web3 AI

June 27, 2025

CARV’s New Roadmap Signals Next Wave Of Web3 AI

June 27, 2025

Bybit Expands Global Reach With Credit Card Crypto Purchases In 25+ Currencies And Cashback Rewards

June 27, 2025

Crypto Flexs is a Professional Cryptocurrency News Platform. Here we will provide you only interesting content, which you will like very much. We’re dedicated to providing you the best of Cryptocurrency. We hope you enjoy our Cryptocurrency News as much as we enjoy offering them to you.

Contact Us : Partner(@)Cryptoflexs.com

Top Insights

Elon Musk, SpaceX And Crypto Hype: What’s Really Going On?

June 28, 2025

Checkpoint #4: Berlinterop | Ether Leeum Foundation Blog

June 28, 2025

TRON Price Propects USDT supply exceeded $ 80 billion

June 28, 2025
Most Popular

PI Coin Price Target $ 2: Is PI Network still in the early stages?

May 28, 2025

Polygon announced the release of Polygon Miden Alpha Testnet v2.

June 8, 2024

US spot Bitcoin ETFs broke their four-day streak of inflows yesterday, recording a net outflow of $52.8 million.

September 19, 2024
  • Home
  • About Us
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms and Conditions
© 2025 Crypto Flexs

Type above and press Enter to search. Press Esc to cancel.