All Topics
All Topics
Technology
Technology
AI
AI
Business
Business
Entertainment
Entertainment
News
News
Programming
Programming
Security
Security
Science
Science
Design
Design
Environment
Environment
Finance
Finance
Crypto
Crypto
Politics
Politics
Sports
Sports
Education
Education
Gaming
Gaming
Art
Art
Music
Music
Health
Health
Books
Books
Food
Food
Travel
Travel
Personal
Personal
Bluesky
Twitter

AI Crawl Control - Crawler drilldowns with extended actions menu

7mo ago

Source

CloudflareAI Crawl Control - Crawler drilldowns with extended actions menucloudflare.com
Snippet from the RSS feed
AI Crawl Control now supports per-crawler drilldowns with an extended actions menu and status code analytics. Drill down into Metrics, Cloudflare Radar, and Security Analytics, or export crawler data for use in WAF custom rules , Redirect Rules , and robots.txt files. What's new Status code distribution chart The Metrics tab includes a status code distribution chart showing HTTP response codes (2xx, 3xx, 4xx, 5xx) over time. Filter by individual crawler, category, operator, or time range to analyze how specific crawlers interact with your site. Extended actions menu Each crawler row includes a three-dot menu with per-crawler actions: View Metrics — Filter the AI Crawl Control Metrics page to the selected crawler. View on Cloudflare Radar — Access verified crawler details on Cloudflare Radar. Copy User Agent — Copy user agent strings for use in WAF custom rules, Redirect Rules, or robots.txt files. View in Security Analytics — Filter Security Analytics by detection IDs (Bot Management customers). Copy Detection ID — Copy detection IDs for use in WAF custom rules (Bot Management customers). Get started Log in to the Cloudflare dashboard, and select your account and domain. Go to AI Crawl Control > Metrics to access the status code distribution chart. Go to AI Crawl Control > Crawlers and select the three-dot menu for any crawler to access per-crawler actions. Select multiple crawlers to use bulk copy buttons for user agents or detection IDs. Learn more about AI Crawl Control .

You might also wanna read

Cloudflare expands AI bot management tools with granular traffic controls for all customers

Cloudflare is celebrating the second "Content Independence Day" by expanding AI traffic management options for all website owners. Building

Cloudflare·1d ago

Cloudflare expands AI bot management tools with granular traffic controls for all customers

Cloudflare is celebrating the second "Content Independence Day" by expanding AI traffic management options for all website owners. Building

blog.cloudflare.com·1d ago

Cloudflare to automatically block web crawlers that collect content for AI companies

Cloudflare announced it will automatically block mixed-use web crawlers that serve AI companies, giving website owners more control over how

engadget.com·13h ago

Cloudflare to automatically block web crawlers that collect content for AI companies

Cloudflare announced it will automatically block mixed-use web crawlers that serve AI companies, giving website owners more control over how

Engadget·13h ago

AI Web Crawlers Account for 30% of Global Web Traffic According to Cloudflare Data

AI web crawlers are consuming a significant portion of global web traffic, with Cloudflare reporting 30% of traffic coming from bots and Fas

theregister.com·10mo ago

Cloudflare Introduces Content Signals Policy for Enhanced Web Content Control

Cloudflare is launching a new Content Signals Policy that enhances robots.txt functionality to give website operators more control over how

blog.cloudflare.com·9mo ago

Cloudflare Launches Agent-Ready Scanner to Check Website AI Compatibility

Cloudflare has launched an Agent-Ready Scanner tool that analyzes websites for AI compatibility by checking standards like robots.txt, MCP,

Product Hunt·2mo ago

Comments

Sign in to join the conversation.

No comments yet. Be the first.