Databricks Open Sources Dicer Auto-Sharding System for Scalable Services
By
vivek-jain
4mo ago· 15 min readen
100/100
Golden Brown
Bagelometer↗
Pure flour-power. Hearty enough to carry you through lunch.
Score100Typepress releaseSentimentpositive
Summary
Databricks announces the open sourcing of Dicer, their auto-sharding system that dynamically manages sharding assignments to enable low latency, scalable, and reliable sharded services. Dicer is foundational to Databricks' products, improving performance, fleet efficiency, and reducing cloud costs by handling restarts, failures, and shifting workloads automatically.
Key quotes
· 3 pulledToday, we are excited to announce the open sourcing of one of our most critical infrastructure components, Dicer: Databricks' auto-sharder, a foundational system designed to build low latency, scalable, and highly reliable sharded services.
It is behind the scenes of every major Databricks product, enabling us to deliver a consistently fast user experience while improving fleet efficiency and reducing cloud costs.
Dicer achieves this by dynamically managing sharding assignments to keep services responsive and resilient even in the face of restarts, failures, and shifting workloads.
Open sourcing Dicer, Databricks’ auto sharder. Learn why auto sharding matters, how Dicer works, and how it keeps Unity Catalog and SQL fast through restarts.
