Site Blocks Outdated Browsers to Combat LLM Training Crawlers
By
todsacerdoti
Sesame, salt, and substance. A flagship bake.
Summary
The article explains that users are seeing an error message because they're using outdated browsers that the site owner's anti-crawler system flags as suspicious. The site owner is blocking old browser versions (particularly Chrome) to combat a surge of high-volume crawlers in early 2025 that use outdated user agents, likely for LLM training data collection. This is an experimental measure to reduce server load on the Wandering Thoughts blog and CSpace wiki.
Key quotes
· 3 pulledYou're using a browser version that my anti-crawler precautions consider suspicious, most often because it's too old (most often this applies to versions of Chrome)
As of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that use a variety of old browser user agents, especially Chrome user agents
To reduce the load on Wandering Thoughts I'm experimenting with (attempting to) block all of them, and you've run into this
You might also wanna read
Website Implements Anubis Proof-of-Work System to Protect Against AI Scraping
The article explains that the website uses Anubis, a Proof-of-Work system similar to Hashcash, to protect against AI companies aggressively
Website Implements Anubis Proof-of-Work System to Protect Against AI Scraping
The article explains that the website is using Anubis, a Proof-of-Work system similar to Hashcash, to protect against AI companies aggressiv
Website Implements Anubis Proof-of-Work System to Protect Against AI Scraping
The article explains that the website is using Anubis, a Proof-of-Work system similar to Hashcash, to protect against AI companies aggressiv
Site Blocks Outdated Browsers to Combat LLM Training Crawlers
The article explains that users are seeing an error message because they're using outdated browsers that the site owner's anti-crawler syste
Website Blocks Outdated Browsers to Combat LLM Training Crawlers
A website owner explains that visitors are seeing an error message because their browsers are being blocked by anti-crawler measures. The si
Mediacom launches 5-Gig residential internet service across 9 states
Mediacom has launched a new 5-Gig residential internet service, its fastest speed tier ever, now available to over 500,000 homes across mult
gigcitygeek.com·1d ago