Website Blocks Old Browsers to Combat LLM Training Crawlers
By
ingve
A five-star bake. Worth schmearing, sharing, saving.
Summary
A website owner explains that visitors are seeing an error message because their browsers are being blocked by anti-crawler measures. The site is implementing these blocks due to a surge of high-volume crawlers in early 2025, many using old browser user agents (particularly Chrome versions) to scrape content, likely for LLM training. The message is an apology and explanation to legitimate users who encounter this block.
Key quotes
· 3 pulledUnfortunately you're using a browser version that my anti-crawler precautions consider suspicious, most often because it's too old (most often this applies to versions of Chrome)
as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that use a variety of old browser user agents, especially Chrome user agents
To reduce the load on Wandering Thoughts I'm experimenting with (attempting to) block all of them, and you've run into this
You might also wanna read
Understanding WebAuthn credential protection policy and discoverable credentials
This article explains the WebAuthn credential protection policy, specifically how developers can use the `residentKey` option to control whe
Let's Encrypt's Challenge: Creating Intentionally Broken Certificates for Testing
Let's Encrypt, as a Certificate Authority, faces unique challenges in testing certificate validation systems. While most tools focus on main
Website Uses Anubis Proof-of-Work System to Protect Against AI Scraping
The article explains that the website uses Anubis, a Proof-of-Work system similar to Hashcash, to protect against AI companies aggressively
Firefox 148 Introduces Standardized Sanitizer API for Enhanced XSS Protection
Firefox 148 introduces the standardized Sanitizer API as a security enhancement to protect against cross-site scripting (XSS) attacks. The n
Website Implements Anubis Proof-of-Work System to Block AI Scraping
The article explains that the website is using Anubis, a Proof-of-Work system similar to Hashcash, to protect against AI companies aggressiv
Website Implements Anubis Proof-of-Work System to Block AI Scraping
This article explains that the website is using Anubis, a Proof-of-Work system similar to Hashcash, to protect against AI companies aggressi