FeedBagel

All Topics

Art

AI Crawl Control - Introducing Redirects for AI Training

2mo ago

Source

CloudflareAI Crawl Control - Introducing Redirects for AI Trainingcloudflare.com

Snippet from the RSS feed

Cloudflare's network now supports redirecting verified AI training crawlers to canonical URLs when they request deprecated or duplicate pages. When enabled via AI Crawl Control > Quick Actions , AI training crawlers that request a page with a canonical tag pointing elsewhere receive a 301 redirect to the canonical version. Humans, search engine crawlers, and AI Search agents continue to see the original page normally. This feature leverages your existing tags. No additional configuration required beyond enabling the toggle. Available on Pro, Business, and Enterprise plans at no additional cost. Refer to the Redirects for AI Training documentation for details.

You might also wanna read

Cloudflare to automatically block web crawlers that collect content for AI companies

Cloudflare announced it will automatically block mixed-use web crawlers that serve AI companies, giving website owners more control over how

Engadget·11h ago

Cloudflare to automatically block web crawlers that collect content for AI companies

Cloudflare announced it will automatically block mixed-use web crawlers that serve AI companies, giving website owners more control over how

engadget.com·11h ago

Cloudflare expands AI bot management tools with granular traffic controls for all customers

Cloudflare is celebrating the second "Content Independence Day" by expanding AI traffic management options for all website owners. Building

Cloudflare·1d ago

Cloudflare expands AI bot management tools with granular traffic controls for all customers

Cloudflare is celebrating the second "Content Independence Day" by expanding AI traffic management options for all website owners. Building

blog.cloudflare.com·1d ago

Cloudflare introduces pay-per-crawl for AI bots

blog.cloudflare.com·1y ago

AI Web Crawlers Account for 30% of Global Web Traffic According to Cloudflare Data

AI web crawlers are consuming a significant portion of global web traffic, with Cloudflare reporting 30% of traffic coming from bots and Fas

theregister.com·10mo ago

Cloudflare to Block AI Bots and Seek Google's Help to Block AI Overviews

Cloudflare announced plans to block AI bots by default and introduce a pay per crawl initiative to compensate for content consumed by AI. CE

seroundtable.com·1y ago

Cloudflare Accuses Perplexity of Circumventing Website Restrictions with AI Crawlers

Cloudflare accuses AI search startup Perplexity of bypassing website restrictions by concealing its AI crawlers' identities to access blocke

The Verge·11mo ago

Comments

No comments yet. Be the first.