Site Blocks Outdated Browsers to Combat LLM Training Crawlers
You're probably reading this page because you've attempted to access some part of my blog (Wandering Thoughts) or CSpace, the wiki thing it's part of. Unfortunately you're using a browser version…
Read the full articleYou might also wanna read
Cloudflare to automatically block web crawlers that collect content for AI companies
The hosting platform wants sites to have more control over how AI companies use their content.
Engadget·13d agoCloudflare to automatically block web crawlers that collect content for AI companies
The hosting platform wants sites to have more control over how AI companies use their content.
engadget.com·13d ago
News websites blocking Wayback Machine to prevent AI scraping threatens web archiving
This isn't the first time the Wayback Machine has faced what could be deemed an existential threat.
techradar.com·1mo ago
23 major news sites block Internet Archive's Wayback Machine crawler, analysis finds
An analysis from Originality AI, an AI detection company, found that 23 big news sites block the Internet Archive’s crawler, Wired reports.
theverge.com·1mo ago
Cloudflare will block AI training crawlers by default on ad-supported sites
The company says new controls will let publishers separate search, agent use and model training The post Cloudflare will block AI training c
Cloudflare to block AI web crawlers by default, shifting to opt-in model for sites
Cloudflare will begin blocking mixed-use AI web crawlers by default, giving website owners greater control over how their content is used.
dataconomy.com·7d ago.png)
AI user-agents, bots, and crawlers to watch (November 2025 update)
November 2025 list of AI user-agents, with practical robots.txt and how to set up for an audit. LLM/AI crawlers leave their signatures throu
Comments
Sign in to join the conversation.
No comments yet. Be the first.