Pipelines, R2, Workers - Cloudflare Pipelines now available in beta

Cloudflare Pipelines is now available in beta, to all users with a Workers Paid plan. Pipelines let you ingest high volumes of real time data, without managing the underlying infrastructure. A single pipeline can ingest up to 100 MB of data per second, via HTTP or from a Worker . Ingested data is automatically batched, written to output files, and delivered to an R2 bucket in your account. You can use Pipelines to build a data lake of clickstream data, or to store events from a Worker. Create your first pipeline with a single command: $ npx wrangler@latest pipelines create my-clickstream-pipeline --r2-bucket my-bucket 🌀 Authorizing R2 bucket "my-bucket" 🌀 Creating pipeline named "my-clickstream-pipeline" ✅ Successfully created pipeline my-clickstream-pipeline Id: 0e00c5ff09b34d018152af98d06f5a1xvc Name: my-clickstream-pipeline Sources: HTTP: Endpoint: Authentication: off Format: JSON Worker: Format: JSON Destination: Type: R2 Bucket: my-bucket Format: newline-delimited JSON Compression: GZIP Batch hints: Max bytes: 100 MB Max duration: 300 seconds Max records: 100,000 🎉 You can now send data to your pipeline! Send data to your pipeline's HTTP endpoint: curl " -d '[{ ...JSON_DATA... }]' To send data to your pipeline from a Worker, add the following configuration to your config file: { "pipelines": [ { "pipeline": "my-clickstream-pipeline", "binding": "PIPELINE" } ] } Head over to our getting started guide for an in-depth tutorial to building with Pipelines.