news.ycombinator.com1mo ago
The article content is not actual article content but rather an HTTP 400 error message indicating that a request has been blocked by server security policies. The message advises contacting support if the user believes this is an error.
The article explains that users are seeing an error message because they're using outdated browsers that the site owner's anti-crawler system flags as suspicious. The site owner is blocking old browser versions (particularly Chrome) to combat a surge of high-volume crawlers in early 2025 that use outdated user agents, likely for LLM training data collection. This is an experimental measure to reduce server load on the Wandering Thoughts blog and CSpace wiki.
The article explains that the website uses Anubis, a Proof-of-Work system similar to Hashcash, to protect against AI companies aggressively scraping websites. This protection is implemented because mass scraping can cause server downtime and make resources inaccessible. Anubis wo
blog.miguelgrinberg.com5mo agoThe article provides a technical guide on implementing an nginx configuration to protect a Forgejo instance from AI web crawlers while maintaining accessibility for legitimate users. The solution uses a cookie-based system that blocks crawlers by default but allows access to users with a specific cookie or using git-related user agents. The configuration ret
her.esy.fun5mo ago
benjojo.co.uk5mo ago
