Cloudflare introduces pay-per-crawl for AI bots
By
scotchmi_st
Article URL: https://blog.cloudflare.com/introducing-pay-per-crawl/
Comments URL: https://news.ycombinator.com/item?id=44432385
Points: 20
# Comments: 0
You might also wanna read
Cloudflare Accuses Perplexity of Circumventing Website Restrictions with AI Crawlers
Cloudflare accuses AI search startup Perplexity of bypassing website restrictions by concealing its AI crawlers' identities to access blocke
The Verge·10mo ago
IAB Tech Lab releases AI bot and crawler management guidance for public comment
IAB Tech Lab has released guidance on AI system bot and crawler management strategies, opening a public comment period through June 25, 2026
ppc.land·21h ago
Cloudflare Launches Agent-Ready Scanner to Check Website AI Compatibility
Cloudflare has launched an Agent-Ready Scanner tool that analyzes websites for AI compatibility by checking standards like robots.txt, MCP,
Product Hunt·1mo ago
Tool to Check Website Crawlability by ChatGPT and Other AI Models
This article describes a tool called "Check if your website can get crawled by ChatGPT" that helps website owners test whether their sites c
Product Hunt·10mo ago
HasData launches managed web scraping service for AI agents with pay-per-success billing
HasData is a managed web scraping service designed for AI agents and data pipelines. It handles infrastructure challenges like proxies, brow
Product Hunt·24d ago