Building blobd: A High-Performance Single-Machine Object Store with Sub-Millisecond Reads
By
charlieirish
Pulled from the oven just right. Trustworthy, fact-dense, deeply satisfying.
Summary
The article details the development of 'blobd', a single-machine object store optimized for low-latency random reads and high-throughput uploads. Built using async Rust, io_uring, and atomic writes, the system achieves sub-millisecond reads and 15 GB/s upload speeds by leveraging modern NVMe SSD capabilities. The author explores technical challenges like metadata management, data layout optimization, and performance benchmarking against existing solutions, presenting this as both a learning exercise and a demonstration of what's possible with current hardware.
Key quotes
· 5 pulledModern NVMe SSDs can do hundreds of thousands of random I/O reads per second.
I decided to experiment with writing an object store from scratch as a fun learning exercise, and to see how much I could get out of NVMe disks I had on my bare metal machines.
Specifically, I wanted to really optimize for random reads and small objects — serving user content where the lower the latency, the better.
This would not focus on other features; S3 is bottomless, distributed, and managed for example.
Deep dive into blobd, an object store designed for constant low latency reads using async Rust, io_uring and atomic writes.
You might also wanna read
Blober.io: A Local-First Workflow Engine for Managing Files Across Cloud Storage Providers
Blober.io is a local-first workflow engine designed to help developers and teams manage large-scale data transfers between different cloud s
Product Hunt·5mo ago
Bitfield: A plug-in runtime and database claiming 0.69ns read and 0.58ns write speeds
Bitfield is a plug-in runtime and database marketed as the world's fastest, with claims of 0.69ns/read and 0.58ns/write performance. It targ
Product Hunt·23d ago