Abstract
In modern database systems, durability guarantees are crucial but often insufficient in scenarios involving extended system outages or data corruption. Additionally, challenges arise when a single user mutation request is distributed across different database nodes or across multiple databases, necessitating a robust mechanism for mutation replay.
At Netflix, we have developed an innovative architecture that not only addresses these issues but also enhances the system's overall functionality. This talk will outline how we leverage Write-Ahead Logging (WAL) as a service to ensure all database mutations are durable and maintain ordering guarantees, even during periods when a downstream database is temporarily unavailable.
A key design principle in our approach is the separation of concerns, which helps manage system complexity and facilitates higher adoption. By employing a pluggable architecture, we enable the development of high-leverage capabilities such as secondary indexes, delayed queues, and generic replication services. These capabilities ensure data consistency across various non-distributed data stores like RocksDB, Redis, and Memcached.
This talk explores the key considerations that must be taken into account when building WAL for various use cases.
Key Takeaways
- How building systems with pluggable architecture helps in higher adoption and building high-leverage capabilities.
- Keeping separation of concerns as a core design principle to manage the complexity of the system.
- How to invest in building newer systems/architectures as the use cases evolve.
Speaker

Prudhviraj Karumanchi
Staff Software Engineer at Data Platform @Netflix, Building Large-Scale Distributed Storage Systems and Cloud Services, Previously @Oracle, @NetApp, and @EMC/Dell
Prudhviraj Karumanchi is a Staff Software Engineer at Data Platform@Netflix, building large-scale distributed storage systems and cloud services. Prudhvi is currently leading the Netflix caching infrastructure. Prior to Netflix, Prudhvi worked at large enterprises such as Oracle, NetApp, and EMC/Dell, building cloud infrastructure and contributing to File, Block, and Object storage systems
Find Prudhviraj Karumanchi at:
Speaker

Vidhya Arvind
Tech Lead @Netflix Data Platform, Founding Member of Data Abstractions at Netflix, Previously @Box and @Verizon
Vidhya Arvind is a Tech Lead at Netflix and a founding architect of Netflix’s cutting-edge data abstraction platform. She is a recognized expert in designing and delivering scalable, high-impact data abstractions that empower thousands of developers across the organization to move faster with confidence. With expertise in crafting robust APIs and high-performance abstractions, Vidhya drives the seamless operation of complex abstractions at massive scale. She is known for her strategic thinking, curiosity, and a systems-level mindset that fuels her passion for debugging, innovating, and solving deeply technical challenges. Vidhya has played a pivotal role in shaping the evolution of Netflix's data infrastructure, enabling mission-critical systems to run with exceptional efficiency, reliability, and resilience.