The Case Against Blocking LLM Crawlers on Websites
By
johnjwang
9mo ago· 2 min readenOpinion
65/100
Toasty
Bagelometer↗
Lightly browned and well buttered. A solid pick from the rack.
Score65TypeopinionSentimentneutral
Summary
The article argues against blocking large-language-model (LLM) crawlers from websites, comparing it to allowing Google to index content. It critiques the moral and practical opposition to AI scraping, emphasizing the value of content visibility.
Key quotes
· 3 pulledBut how many of you wouldn’t hook up your website to Google?
I know one of the primary reasons that I do anything online is to provide an outlet for someone else to see it.
They’re generally highly vitriolic, with people opposing this on both moral grounds ('AI is stealing your content') as well as displaying a general distaste for AI.
Perplexity was recently accused of scraping sites that had explicitly disallowed LLM crawlers in their robots.txt files. In the wake of that revelation, a wave of how-to guides for blocking…
You might also wanna read
Google adds llms.txt to Chrome Lighthouse agentic audits, sparking debate on AI web standard adoption
Google has added llms.txt to its Chrome Lighthouse toolset under a new "Agentic browsing audits" section, signaling that the company now con
ppc.land·2d ago
Website Uses Anubis Proof-of-Work System to Block AI Scrapers
This article describes a website protection system called Anubis that uses a Proof-of-Work scheme (similar to Hashcash) to defend against ag
lkml.org·5d ago