AI Crawl Control - Pay Per Crawl (Private beta) - Discovery API, custom pricing, and advanced configuration

Pay Per Crawl is introducing enhancements for both AI crawler operators and site owners, focusing on programmatic discovery, flexible pricing models, and granular configuration control. For AI crawler operators Discovery API A new authenticated API endpoint allows verified crawlers to programmatically discover domains participating in Pay Per Crawl. Crawlers can use this to build optimized crawl queues, cache domain lists, and identify new participating sites. This eliminates the need to discover payable content through trial requests. The API endpoint is GET and requires Web Bot Auth authentication. Refer to Discover payable content for authentication steps, request parameters, and response schema. Payment header signature requirement Payment headers ( crawler-exact-price or crawler-max-price ) must now be included in the Web Bot Auth signature-input header components. This security enhancement prevents payment header tampering, ensures authenticated payment intent, validates crawler identity with payment commitment, and protects against replay attacks with modified pricing. Crawlers must add their payment header to the list of signed components when constructing the signature-input header . New crawler-error header Pay Per Crawl error responses now include a new crawler-error header with 11 specific error codes for programmatic handling. Error response bodies remain unchanged for compatibility. These codes enable robust error handling, automated retry logic, and accurate spending tracking. For site owners Configure free pages Site owners can now offer free access to specific pages like homepages, navigation, or discovery pages while charging for other content. Create a Configuration Rule in Rules > Configuration Rules , set your URI pattern using wildcard, exact, or prefix matching on the URI Full field, and enable the Disable Pay Per Crawl setting. When disabled for a URI pattern, crawler requests pass through without blocking or charging. Some paths are always free to crawl. These paths are: /robots.txt , /sitemap.xml , /security.txt , /.well-known/security.txt , /crawlers.json . Get started AI crawler operators : Discover payable content