AI Crawl Control - New Robots.txt tab for tracking crawler compliance

AI Crawl Control now includes a Robots.txt tab that provides insights into how AI crawlers interact with your robots.txt files. What's new The Robots.txt tab allows you to: Monitor the health status of robots.txt files across all your hostnames, including HTTP status codes, and identify hostnames that need a robots.txt file. Track the total number of requests to each robots.txt file, with breakdowns of successful versus unsuccessful requests. Check whether your robots.txt files contain Content Signals directives for AI training, search, and AI input. Identify crawlers that request paths explicitly disallowed by your robots.txt directives, including the crawler name, operator, violated path, specific directive, and violation count. Filter robots.txt request data by crawler, operator, category, and custom time ranges. Take action When you identify non-compliant crawlers, you can: Block the crawler in the Crawlers tab Create custom WAF rules for path-specific security Use Redirect Rules to guide crawlers to appropriate areas of your site To get started, go to AI Crawl Control > Robots.txt in the Cloudflare dashboard. Learn more in the Track robots.txt documentation .

AI Crawl Control - New Robots.txt tab for tracking crawler compliance

Source

You might also wanna read

Cloudflare to automatically block web crawlers that collect content for AI companies

Cloudflare to automatically block web crawlers that collect content for AI companies

Cloudflare expands AI bot management tools with granular traffic controls for all customers

Cloudflare expands AI bot management tools with granular traffic controls for all customers

Tool to Check Website Crawlability by ChatGPT and Other AI Models

Revisiting the Impact of Robots.txt on My Blog: Lessons Learned

Known Agents

BotCost.dev: A Free Browser-Based Tool to Detect and Block AI Bot Traffic on Your Website

Comments