AI Scrapers Attempt to Access Commented-Out Scripts, Revealing Bot Behavior Patterns
By
ColinWright
The kind of bagel that ruins lesser bagels for you.
Summary
A developer discovered abusive bot behavior when AI scrapers attempted to access a commented-out JavaScript file that didn't exist on the server, revealing a new method for identifying poorly-behaved bots. The article details how the author accidentally deployed a page with a commented script tag, leading to 404 errors when AI scrapers tried to fetch the non-existent file. This incident exposes how some AI training data collection bots ignore HTML comments and attempt to scrape any referenced resources, regardless of whether they're actually part of the page content.
Key quotes
· 5 pulledAI scrapers request commented scripts
I discovered some abusive bot behaviour during a routine follow-up on anomalies that had shown up in my server's logfiles
There were a bunch of 404 errors ('Not Found') for a specific JavaScript file
I accidentally committed and deployed a commented-out script tag that I'd included in the page while prototyping a new feature
A new avenue for identifying greedy, badly-behaved bots
You might also wanna read
Hackers Abuse AI Chatbot Recommendations to Push Malicious Software Download Links
Website Uses Anubis Proof-of-Work System to Block AI Scrapers
This article describes a website protection system called Anubis that uses a Proof-of-Work scheme (similar to Hashcash) to defend against ag
Agent Monitor: Server-Side Analytics Tool Reveals 65% of Web Traffic is Bots, 24% AI Bots
Agent Monitor is a server-side analytics tool that captures and classifies AI and bot traffic, revealing that 65% of traffic across 249 site
Product Hunt·3mo ago
Website uses Anubis Proof-of-Work system to block AI scraping bots
This article describes a website protection system called Anubis, which uses a Proof-of-Work scheme (similar to Hashcash) to defend against
How hackers exploit AI chatbot personalities through prompt injection attacks
This article discusses how hackers are exploiting AI chatbot "personalities" through prompt injection and jailbreaking techniques. Initially
The Verge·7d ago
Website uses Anubis Proof-of-Work system to block AI scraping bots
This article describes a website protection system called Anubis that uses a Proof-of-Work scheme (similar to Hashcash) to defend against ag