Abstract
“Diskless” systems that delegate durability to object storage are everywhere, and for three good reasons:
- The economics are unbeatable: storage is priced at a fraction of block storage or NVMe, and inter-AZ data-transfers are free.
- Object storage handles replication and provides 99.999999999% durability, solving distributed systems’ hardest problem.
- There are hundreds of competent engineers dedicated to keeping it highly available.
But object storage was built for throughput and offline data processing, not for the access patterns of online systems. Building on it means confronting high per-request latency, managing immutability and navigating per-request API costs.
This talk covers the necessary mental models for working around those limitations and how we applied them to build SlateDB, an OSS object-native LSM engine. We’ll discuss how to effectively batch and buffer writes, strategies for caching to speed up reads and the protocols for building correct systems.
You'll leave with a practical framework for deciding whether object storage is the right durability layer for you, and how to design around its limits.
Speaker
Almog Gavra
Co-Founder @Responsive.dev - Building Object-Native Databases, Previously @Confluent and @LinkedIn
Almog is a Co-Founder at responsive.dev and a top contributor to SlateDB, an OSS object-store native LSM engine. Previously, he built stream processing engines at Confluent and scaled search infrastructure at LinkedIn. He’s a database nerd at heart who loves writing and teaching about the toughest concepts of distributed systems.
