Major News Sites Block Internet Archive's Wayback Machine to Deter AI Scrapers, Threatening Digital History Preservation

Summary

Major news outlets including CNN, The New York Times, and Reuters are blocking the Internet Archive's Wayback Machine from archiving their content, according to an analysis by Originality AI. This move is primarily aimed at preventing AI companies from scraping their content for training large language models, but it also threatens to create gaps in the digital historical record. The Wayback Machine, which has archived over one trillion web pages since its founding nearly 30 years ago, is being blocked via robots.txt files — the same protocol originally designed to prevent server overload, not to restrict access to digital history. The article explores the tension between publishers' legitimate concerns about AI scraping and the unintended consequences of blocking a vital preservation tool that serves journalists, researchers, courts, and the public.