Cloudflare data shows AI bots crawl sites up to 50,000 times per human visitor as regulatory pushback grows
Cloudflare shows AI bots crawling sites 50,000 times per visitor as Munich strips Google's cover for Overviews and Google dismisses a rival crawler standard.
Read the full articleYou might also wanna read
AI Web Crawlers Account for 30% of Global Web Traffic According to Cloudflare Data
Opinion: But the cure may ruin the web....
theregister.com·10mo ago
Bots - New options to manage AI traffic
Not all AI traffic is the same. Now, all customers — including those on the Free plan — can manage AI crawlers based on what they actually d
AI Crawlers from Meta and OpenAI Overwhelm Websites with Excessive Traffic
: One fetcher bot seen smacking a website with 39,000 requests per minute
theregister.com·10mo agoMedia Briefing: Declared ‘good bots,’ mixed-use crawlers, gray scrapers – how AI accesses publisher content
The Cloudflare's latest AI settings reshape how compliant crawlers behave, yet the biggest leakage for publisher content remains a gray scra
Developer Battles AWS-Based Bot Sending 2 Billion Monthly Requests
I have been struggling with a bot– 'Mozilla/5.0 (compatible; crawler)' coming from AWS Singapore – and sending an absurd number of requests
news.ycombinator.com·8mo agoAI Crawl Control - AI Crawl Control
Every site on Cloudflare now has access to AI Audit , which summarizes the crawling behavior of popular and known AI services. You can use t
Comments
Sign in to join the conversation.
No comments yet. Be the first.