Logs, Pipelines - Cloudflare Pipelines as a Logpush destination
2mo ago
Source
CloudflareLogs, Pipelines - Cloudflare Pipelines as a Logpush destinationcloudflare.comLogpush has traditionally been great at delivering Cloudflare logs to a variety of destinations in JSON format. While JSON is flexible and easily readable, it can be inefficient to store and query at scale. With this release, you can now send your logs directly to Pipelines to ingest, transform, and store your logs in R2 as Parquet files or Apache Iceberg tables managed by R2 Data Catalog . This makes the data footprint more compact and more efficient at querying your logs instantly with R2 SQL or any other query engine that supports Apache Iceberg or Parquet. Transform logs before storage Pipelines SQL runs on each log record in-flight, so you can reshape your data before it is written. For example, you can drop noisy fields, redact sensitive values, or derive new columns: INSERT INTO http_logs_sink SELECT ClientIP, EdgeResponseStatus, to_timestamp_micros(EdgeStartTimestamp) AS event_time, upper (ClientRequestMethod) AS method, sha256(ClientIP) AS hashed_ip FROM http_logs_stream WHERE EdgeResponseStatus >= 400 ; Pipelines SQL supports string functions, regex, hashing, JSON extraction, timestamp conversion, conditional expressions, and more. For the full list, refer to the Pipelines SQL reference . Get started To configure Pipelines as a Logpush destination, refer to Enable Cloudflare Pipelines .
You might also wanna read
Cloudflare Launches Fully-Managed Data Platform for Analytical Data on Global Infrastructure
Cloudflare announced the Cloudflare Data Platform, a fully-managed suite of products for ingesting, storing, and querying analytical data. B
WarpStream Tableflow: Converting Kafka Data to Iceberg Tables with Low Latency
WarpStream Tableflow is a new product that addresses the challenges of converting Kafka topic data into Apache Iceberg tables. The article a
warpstream.com·9mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.