Cloudflare Introduces Content Signals Policy for Enhanced Web Content Control

If we want to keep the web open and thriving, we need more tools to express how content creators want their data to be used while allowing open access.

Today the tradeoff is too limited. Either website operators keep their content open to the web and risk people using it for unwanted purposes, or they move their content behind logins and limit their audience.

To address the concerns our customers have today about how their content is being used by crawlers and data scrapers, we are launching the Content Signals Policy.

This policy is a new addition to robots.txt that allows website operators more granular control over content usage.