AI Crawlers from Meta and OpenAI Overwhelm Websites with Excessive Traffic
By
rntn
Kettled twice. Extra chewy, extra trustworthy.
Summary
A Fastly report reveals that AI crawlers and fetchers are overwhelming websites with excessive traffic, accounting for 80% of all AI bot traffic. Meta's AI division is responsible for over half of these crawlers, while OpenAI dominates the fetcher category. The bots can make thousands of requests per minute to single websites, with one extreme case showing 39,000 requests per minute, potentially causing significant strain on web infrastructure.
Key quotes
· 4 pulledAI crawlers are putting a heavy load on the open web, slurping up sites at a rate that accounts for 80 percent of all AI bot traffic
Facebook owner Meta's AI division accounts for more than half of those crawlers, while OpenAI accounts for the overwhelming majority
Bots and fetchers can hit websites hard, demanding data from a single site in thousands of requests per minute
One fetcher bot seen smacking a website with 39,000 requests per minute
You might also wanna read
Agent Monitor: Server-Side Analytics Tool Reveals 65% of Web Traffic is Bots, 24% AI Bots
Agent Monitor is a server-side analytics tool that captures and classifies AI and bot traffic, revealing that 65% of traffic across 249 site
Product Hunt·3mo ago
Cloudflare Accuses Perplexity of Circumventing Website Restrictions with AI Crawlers
Cloudflare accuses AI search startup Perplexity of bypassing website restrictions by concealing its AI crawlers' identities to access blocke
The Verge·10mo ago
The Growth of AI Content Spam and the Challenges of the Bot Economy
AI content spam is on the rise, with projections indicating that 90% of web content will be AI-generated by 2025. The growth of the AI spam
Prototypr·2y ago
Free AI SEO Auditor: Open-Source Tool Scores Websites for AI Search Engine Visibility
A review of Free AI SEO Auditor, an open-source tool that scores websites for visibility in AI search engines like ChatGPT, Claude, and Perp
Product Hunt·20d ago