Website Blocks Old Browsers to Combat LLM Training Crawlers in 2025
By
walterbell
The bagel they save for the regulars. Don't skim, savour.
Summary
A website owner explains that visitors are seeing an error message because their browsers are being blocked by anti-crawler measures. The site is implementing these blocks to combat a surge of high-volume crawlers in early 2025, many of which use old browser user agents (particularly Chrome versions) to scrape content, likely for LLM training data collection. The message is directed at legitimate users who encounter this block while trying to access the blog 'Wandering Thoughts' or its associated wiki 'CSpace.'
Key quotes
· 3 pulledUnfortunately you're using a browser version that my anti-crawler precautions consider suspicious, most often because it's too old (most often this applies to versions of Chrome).
Unfortunately, as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that use a variety of old browser user agents, especially Chrome user agents.
To reduce the load on Wandering Thoughts I'm experimenting with (attempting to) block all of them, and you've run into this.
You might also wanna read
Website uses Anubis Proof-of-Work system to block AI scraping bots
This article describes a website protection system called Anubis, which uses a Proof-of-Work scheme (similar to Hashcash) to defend against
Website Uses Anubis Proof-of-Work System to Block AI Scrapers
This article describes a website protection system called Anubis that uses a Proof-of-Work scheme (similar to Hashcash) to defend against ag
Website uses Anubis Proof-of-Work system to block AI scraping bots
This article describes a website protection system called Anubis that uses a Proof-of-Work scheme (similar to Hashcash) to defend against ag