TL;DR
Databricks has open sourced Dicer, its internal auto-sharding control plane used to build low-latency, highly available sharded services. Dicer dynamically manages shard assignments (slices) to avoid downtime, mitigate hot keys, and keep in-memory caches effective across restarts and scaling events.
What happened
Databricks published an engineering post announcing the open source release of Dicer, the company’s auto-sharder used to run production sharded services. Dicer replaces brittle static sharding and the cost-heavy stateless model by continuously computing assignments of key ranges (slices) to application pods, reacting to signals such as load, health, and termination notices. Applications integrate via a client library called the Slicelet, which keeps a local, eventually consistent cache of assignments and reports per-key load summaries back to the Dicer service. Dicer can split, merge, replicate, and reassign slices and isolate hot keys by promoting them to their own slices and replicating them across pods. Databricks says Dicer powers mission-critical components including Unity Catalog and a SQL query orchestration engine, and has kept cache hit rates above 90% during pod restarts while eliminating availability dips.
Why it matters
- Reduces the operational and latency costs associated with stateless services that re-read state on every request.
- Addresses availability problems that arise from static sharding, such as downtime during restarts, split-brain scenarios, and unhandled hot keys.
- Enables dynamic rebalancing and replication of hot key workloads to prevent single-pod overloads.
- Provides a control plane that can improve fleet efficiency and resilience for large-scale, stateful services.
Key facts
- Databricks announced the open source release of Dicer on January 13, 2026.
- Authors of the post include Atul Adya, Colin Meek, Jonathan Ellithorpe, Vivek Jain, and Yongxin Xu.
- Dicer models keys as hashed SliceKeys and manages contiguous ranges called Slices, assigning them to Resources (pods).
- Applications use a library called the Slicelet to watch and cache assignments and to report aggregated per-key load back to Dicer.
- Assignments observed by Slicelets are intentionally eventually consistent, prioritizing availability and fast recovery.
- Dicer dynamically splits, merges, replicates, and reassigns slices and can isolate and replicate hot keys across multiple pods.
- The system is already used in production at Databricks for components such as Unity Catalog and a SQL query orchestration engine.
- Databricks reports Dicer eliminated availability dips and maintained cache hit rates above 90% during pod restarts for those systems.
- Dicer builds on prior academic and engineering work in auto-sharding such as Centrifuge, Slicer, and Shard Manager.
What to watch next
- A promised technical deep dive into Dicer’s design and architecture in a future Databricks post.
- Planned support for stronger consistency guarantees in future releases, referenced as a possible enhancement beyond the current eventually consistent model.
- not confirmed in the source
Quick glossary
- Slice: A contiguous range of hashed keys (SliceKeys) that together span an application's keyspace; the unit of assignment in Dicer.
- SliceKey: A hash-derived representation of an application key used to map keys into slices for sharding.
- Slicelet: A client-side library that applications use to fetch and cache Dicer assignments, receive updates, and report aggregated per-key load.
- Auto-sharder: A control-plane system that dynamically assigns and rebalances key ranges across backend resources to handle load and failures.
- Eventual consistency: A consistency model where different nodes may temporarily observe different states but will converge to the same view over time.
Reader FAQ
What is Dicer?
Dicer is Databricks’ auto-sharder control plane that continuously assigns key ranges (slices) to pods to keep sharded services balanced and available.
Which Databricks services use Dicer?
Databricks states Dicer is used for mission-critical systems including Unity Catalog and a SQL query orchestration engine.
Does Dicer provide strong consistency for shard ownership?
No — the current design exposes assignments to applications with eventual consistency; stronger guarantees are planned for a future release.
Is there guidance on installing or experimenting with Dicer?
The blog post says it includes a guide to install and try Dicer, but installation details are not included in the excerpt.
Skip to main content Login Why Databricks Product Solutions Resources About Ready to get started? Get a Demo Contact Us Try Databricks Blog / Engineering / Article Open Sourcing Dicer:…
Sources
- Open sourcing Dicer: Databricks's auto-sharder
- High Performance Ratelimiting at Databricks
- Best practices for reliability | Databricks on AWS
- Best practices for reliability – Azure Databricks
Related posts
- Running Lean at Scale: building a REPL service for distributed theorem proving
- Nogic: VS Code extension visualizes codebases as interactive graphs
- Redd-Archiver: Self-host Reddit archives — 2.38B posts, offline-ready