Reuters and Time Join Publishers Blocking AI Crawlers by Default, Pushing for Licensing Deals
By
Matt G. Southern
Slow-proofed and worth the wait. Worth its weight in flour.
Summary
Reuters and Time have joined a growing number of publishers defaulting to blocking AI crawlers, using allowlists instead of blocklists to control which bots can access their content. The shift, made in May, follows similar moves by People Inc. and The Atlantic. Reuters reports no traffic loss from the change and reduced costs from serving bots, while executives say the added friction is helping push AI companies toward licensing negotiations. The article highlights that robots.txt is insufficient since 30% of AI bot scrapes reportedly don't honor it.
Key quotes
· 4 pulledReuters says the change hasn't cost it traffic, while cutting what it spends serving bots.
Executives credit the added friction with helping push AI companies toward licensing talks.
Robots.txt works only when crawlers choose to honor it.
Digiday cited a Tollbit report finding that 30% of total AI bot scrapes didn't co
You might also wanna read
Over 340 local news outlets block Internet Archive's Wayback Machine over AI scraping concerns
Major newspaper chains including McClatchy, Advance Local, and Tribune Publishing have joined The New York Times, The Guardian, and USA Toda
Cloudflare to Block AI Bots and Seek Google's Help to Block AI Overviews
Cloudflare announced plans to block AI bots by default and introduce a pay per crawl initiative to compensate for content consumed by AI. CE
seroundtable.com·11mo ago
News Publishers Restrict Internet Archive Access Over AI Data Scraping Concerns
News publishers including The Guardian and The New York Times are restricting access to their content in the Internet Archive's Wayback Mach
Major Publishers Launch Really Simple Licensing Standard for AI Content Scraping
Major web publishers including Reddit, Yahoo, Medium, Quora, and People Inc. have announced support for Really Simple Licensing (RSL), a new
The Verge·9mo ago
The Case Against Blocking LLM Crawlers on Websites
The article argues against blocking large-language-model (LLM) crawlers from websites, comparing it to allowing Google to index content. It
johnjianwang.medium.com·10mo ago
Meta Announces AI Licensing Deals with CNN, Fox News, USA Today, and People Inc.
Meta has announced AI licensing partnerships with major media outlets including CNN, Fox News, USA Today, and People Inc., allowing its AI c
The Verge·6mo ago