Why Anubis Bot Protection Is Ineffective Against LLM Scrapers
By
flexagoon
Baker's choice. Dense with flavour, light on filler.
Summary
The article argues against using Anubis, a proof-of-work based bot protection system, specifically for blocking LLM scrapers. It explains that while LLM training companies' scrapers (particularly Anthropic's ClaudeBot) have become more aggressive and disrespectful of robots.txt and other blocking methods, Anubis is ineffective against them. The author contends that Anubis doesn't work for this purpose and suggests alternative approaches for dealing with LLM scrapers rather than relying on this particular solution.
Key quotes
· 4 pulledIn the past years, scrapers operated by LLM training companies have become more relentless.
This seems to mostly apply to Anthropic's ClaudeBot. OpenAI's GPTBot is pretty well documented and easy to block.
This has led to more and more websites using Anubis, a proof-of-work based bot protection solution that requires all visitors to solve a small cryptographic problem on their device before proceeding.
But here's the thing: Anubis doesn't work.
You might also wanna read
Website Uses Anubis Proof-of-Work System to Block AI Scrapers
This article describes a website protection system called Anubis that uses a Proof-of-Work scheme (similar to Hashcash) to defend against ag
Website uses Anubis Proof-of-Work system to block AI scraping bots
This article describes a website protection system called Anubis that uses a Proof-of-Work scheme (similar to Hashcash) to defend against ag
Website uses Anubis Proof-of-Work system to block AI scraping bots
This article describes a website protection system called Anubis, which uses a Proof-of-Work scheme (similar to Hashcash) to defend against
Anubis: A Proof-of-Work System to Protect Websites from AI Scraping Bots
This article describes a website protection system called Anubis that uses a Proof-of-Work scheme (similar to Hashcash) to defend against ag
Website uses Anubis Proof-of-Work system to deter AI scraping bots
This article describes a website protection system called Anubis, which uses a Proof-of-Work scheme (similar to Hashcash) to defend against
Anubis: A Proof-of-Work System to Protect Websites from AI Scraping
This article describes a website protection system called Anubis, which uses a Proof-of-Work scheme (similar to Hashcash) to defend against