Cloudflare to automatically block web crawlers that collect content for AI companies
By
Ian Carlos Campbell
Summary
Cloudflare announced it will automatically block mixed-use web crawlers that serve AI companies, giving website owners more control over how their content is used to train AI models. The platform is introducing a new tool that lets site operators block crawlers that both index content for search and collect data for AI training. This move addresses growing concerns from publishers about unauthorized use of their content by AI companies like OpenAI and Google for training large language models.
Source
EngadgetCloudflare to automatically block web crawlers that collect content for AI companiesengadget.comKey quotes
· 3 pulledCloudflare has announced plans to automatically block mixed-use web crawlers that serve AI companies.
The hosting platform wants sites to have more control over how AI companies use their content.
This move addresses growing concerns from publishers about unauthorized use of their content by AI companies.
You might also wanna read
Cloudflare to Block AI Bots and Seek Google's Help to Block AI Overviews
Cloudflare announced plans to block AI bots by default and introduce a pay per crawl initiative to compensate for content consumed by AI. CE
seroundtable.com·1y agoBots - New options to manage AI traffic
AI Crawl Control - AI Crawl Control
Cloudflare Accuses Perplexity of Circumventing Website Restrictions with AI Crawlers
Cloudflare accuses AI search startup Perplexity of bypassing website restrictions by concealing its AI crawlers' identities to access blocke
The Verge·11mo ago
Cloudflare expands AI bot management tools with granular traffic controls for all customers
Cloudflare is celebrating the second "Content Independence Day" by expanding AI traffic management options for all website owners. Building
Cloudflare·1d agoCloudflare expands AI bot management tools with granular traffic controls for all customers
Cloudflare is celebrating the second "Content Independence Day" by expanding AI traffic management options for all website owners. Building
blog.cloudflare.com·1d ago
AI Web Crawlers Account for 30% of Global Web Traffic According to Cloudflare Data
AI web crawlers are consuming a significant portion of global web traffic, with Cloudflare reporting 30% of traffic coming from bots and Fas
theregister.com·10mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.