Crawlers to Courtrooms: Why publishers say the AI fight is existential

When publishers representing nearly 400 newspapers sued OpenAI and Microsoft this week in a US federal court, the lawsuit was not just another legal skirmish. For the publishing industry, it marked a turning point in a longer, more consequential battle over who controls value in an AI-first internet.Filed in the Southern District of New York, the complaint alleges that OpenAI and Microsoft trained large language models using copyrighted journalism without authorisation or compensation, stripping copyright management information and reproducing publisher content in AI-generated responses. The suit follows earlier action by The New York Times in 2023 and adds momentum to a growing global pushback from news organisations.Behind the courtroom arguments lies a deeper industry concern: that generative AI systems are rapidly becoming the primary interface for information, absorbing journalism’s economic value while sending little or no traffic back to its creators.At a recent WAN-IFRA gathering, publishers framed the imbalance starkly. Journalism, they noted, produces the original signal which is verified, timely, accountable reporting, at an estimated annual global cost of $50 billion. At the same time, the combined market capitalisation of the world’s top AI companies now exceeds $12.5 trillion, much of it built on systems that depend on ingesting existing human-created knowledge.Blocking AI crawlers: leverage or self-erasure?One of the industry’s first defensive responses has been to block AI crawlers. But executives increasingly warn that this tactic may offer leverage without a long-term strategy.“Blocking AI crawlers may offer short-term leverage, but it’s unlikely to be a sustainable long-term strategy,” said Gopa Menon, founder of The Blurr. “Publishers are trying to protect decades of investment in quality journalism, and that concern is valid. However, as AI becomes a primary interface for discovering information, being completely absent from these ecosystems could reduce visibility and relevance over time.”Menon warned that exclusion from AI-generated answers could amount to self-erasure. “Blocking crawlers feels good for about five minutes. Then you realise you’ve just made yourself invisible in the one place your future readers are starting their search,” he said, adding that brands cited in AI Overviews already see meaningfully higher click-through rates than those left out entirely.According to industry estimates cited by a digital business head of a leading English daily, referral patterns have deteriorated steadily. “Earlier, you would get roughly three clicks for every 200–300 links surfaced,” he said. “Now it’s closer to one click for every 800–900 links. The better AI becomes, the fewer clicks publishers receive.”Licensing, lawsuits and the limits of fair useLicensing deals between AI companies and publishers have emerged as one proposed solution, but executives caution against overstating their immediate impact.“Today, these deals are significant for a handful of large global publishers, but they are not yet large enough to transform the economics of publishing,” Menon said. “Most publishers still depend on advertising, subscriptions, events and commerce for the bulk of their revenues. That said, licensing is strategically important because it establishes the principle that quality content has monetary value in the AI ecosystem.”Sanjay Trehan, a digital strategy advisor, said the industry is operating in a legal and ethical grey zone. “AI companies argue fair use because much of the content they crawl is publicly available, but the reality is that they are curating and publishing someone else’s copyrighted work within closed ecosystems,” he said. “That may be legally debatable, but morally it raises serious questions. The only sustainable solution is revenue sharing.”Trehan said the consequences for publishers are already visible. “AI engines satisfy user queries directly, sharply reducing traffic to original news sites. Once a reader’s curiosity is fulfilled, there is little incentive to visit the source, even if it is cited. This traffic loss translates into a direct hit on advertising revenues, while publishers continue to bear the high cost of producing quality, original journalism.”He warned that without compensation, the imbalance will worsen. “Technology cannot be resisted. AI is advancing inexorably and is a remarkable tool for consumers. But if publishers who invest in exclusive, differentiated content are not compensated, the long-term consequence will be a decline in original journalism and an ecosystem flooded with commoditised, recycled information.”From traffic businesses to trust businessesBeyond lawsuits and licensing, publishers argue the AI shift demands a fundamental rethink of business models built around clicks.“It means publishing must stop thinking of itself as a traffic business and start thinking of itself as a trust and sourcing business,” Menon said. “The click was always a means, not the end. We just built two decades of ad models pretending it was the end.”Under this emerging logic, publishing may split into two tiers: originators that break news, conduct investigations and build brands AI systems must cite, and aggregators that are quietly dis-intermediated because AI can replicate their function more efficiently.The digital business head of the English daily framed this as an accountability issue. Courts, he noted, rarely judge journalism, but journalists constantly scrutinise court judgments, government data and official narratives, often forcing institutions to revise their positions. He cited the revision of Covid-era seroprevalence data following sustained media questioning as an example of journalism’s role.“Globally, there are only a few reliable sources of current, fact-based information,” he said. “Without that, large AI models cannot function. Influencers can amplify information, but they cannot do original, evidence-based reporting.”Geographically, responses to AI diverge sharply. In Europe and the United States, an estimated 75–80 percent of leading publishers have blocked OpenAI and other AI crawlers. In India, by contrast, roughly 70 percent of publishers have not, and formal licensing arrangements remain limited.The digital head pointed to OpenAI’s publicly disclosed 2024 licensing agreement with Le Monde, which drew criticism from other publishers and was followed by a decision to stop publicly announcing such deals. Similar agreements may exist, he said, but remain opaque.Some publishers are exploring alternative models. The Economist, for instance, has created separate access routes for AI systems and human readers, effectively treating AI as another paying customer. Whether such approaches can scale across a fragmented industry remains uncertain.What is no longer in doubt is the scale of the stakes. As Trehan put it, “Fair use alone cannot be the end of the conversation. A commercial understanding between AI giants and mainstream publishers is essential, not just for journalism’s survival, but for the long-term quality of the information ecosystem itself.”