AI Crawl Control - Enhanced crawler insights and custom 402 responses

We improved AI crawler management with detailed analytics and introduced custom HTTP 402 responses for blocked crawlers. AI Audit has been renamed to AI Crawl Control and is now generally available. Enhanced Crawlers tab: View total allowed and blocked requests for each AI crawler Trend charts show crawler activity over your selected time range per crawler Custom block responses (paid plans): You can now return HTTP 402 "Payment Required" responses when blocking AI crawlers, enabling direct communication with crawler operators about licensing terms. For users on paid plans, when blocking AI crawlers you can configure: Response code: Choose between 403 Forbidden or 402 Payment Required Response body: Add a custom message with your licensing contact information Example 402 response: HTTP 402 Payment Required Date : Mon, 24 Aug 2025 12:56:49 GMT Content-type : application/json Server : cloudflare Cf-Ray : 967e8da599d0c3fa-EWR Cf-Team : 2902f6db750000c3fa1e2ef400000001 { " message " : "Please contact the site owner for access." }

AI Crawl Control - Enhanced crawler insights and custom 402 responses

Source

You might also wanna read

Cloudflare to automatically block web crawlers that collect content for AI companies

Cloudflare to automatically block web crawlers that collect content for AI companies

Cloudflare expands AI bot management tools with granular traffic controls for all customers

Cloudflare expands AI bot management tools with granular traffic controls for all customers

Cloudflare introduces pay-per-crawl for AI bots

AI Web Crawlers Account for 30% of Global Web Traffic According to Cloudflare Data

Reuters and Time Join Publishers Blocking AI Crawlers by Default, Pushing for Licensing Deals

Cloudflare to Block AI Bots and Seek Google's Help to Block AI Overviews

Comments