Speaker
Abstract
Data recovery is more than just backing up and restoring a data store. The goal of any disaster recovery effort is getting the system back to working as expected across all of its parts. Recovering data by itself only brings us back in time to a view of reality that might not reflect how each part sees the world. This could mean millions of different views of reality in large systems.
This talk covers challenges, patterns, and practices for disaster recovery actions in massively distributed systems. It focuses on two commonly used patterns for restoring the whole system to the same reality:
- Rebuild the world
- Restore & reconcile
We will discuss how these approaches were used in different systems, the challenges and tradeoffs experienced, and why sometimes the answer is "Why not both?" Finally, we’ll explore practices that help improve confidence and recovery time, reducing stress and ensuring things get back to working as fast as possible.
Interview
My work centers around improving the reliability of Google's infrastructure as a service offering. A lot of the work is proactively identifying and mitigating areas of risk in the system, but also, my teams run incident response and drive the learning from the incidents process.
In all the companies I've worked for, there's been a moment where we needed to recover or repair some critical data, and the architectural decision made early on either made that moment a lot easier than it needed to be or a lot harder. I wanted to give folks some tools for reasoning about their architecture's ability to respond to disasters so maybe it will fall on the easy side for them.
This talk is for senior engineers who might be making big architectural decisions and system engineers that might be involved in any disaster-level response.
I would like folks to walk away with the understanding that disaster recovery is more challenging than just backing up your data stores. If you want your system to be recoverable in a disaster, you have to make sure the architecture will support it.
Topics
QCon San Francisco 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Designing for Resilience Hosted by Javier Fernandez-Ivern Staff Software Engineer @Netflix with Over 24 Years in Software EngineeringFrom the same track
Tuesday 3 October
10:35 Ballroom BC Session Architecture Disaster Recovery Across a Million Pieces Michelle Brush Engineering Director, SRE @Google, Previously Director of HealtheIntent Architecture @Cerner Corporation & Lead Engineer @Garmin, Author of "2 out of the 97 Things Every SRE Should Know" Data recovery is more than just backing up and restoring a data store. The goal of any disaster recovery effort is getting the system back to working as expected across all of its parts. 11:45 Ballroom BC Session Reliability Designing Fault-Tolerant Software with Control System Transparency Jon Moore Staff Software Engineer @Stripe with over 35 years of software engineering experience across both academia and industry Teams at NASA and JPL that create mission-critical software for spacecraft take a principled approach to fault tolerance. Let's see how those same principles, centered around a concept of transparency, can help us achieve reliability in pragmatic, modern software delivery settings. 13:35 Ballroom BC Session Resiliency How Do We Talk to Each Other? How Surfacing Communication Patterns in Organizations Can Help You Understand and Improve Your Resilience Nora Jones Founder and CEO @jeli_io, Founder of Learning From Incidents (LFI) Online Community and Conference As a system increases in inevitable complexity, it becomes impossible for a single operator to have a clear, unambiguous understanding of what's happening in the system. Understanding the system requires a joint effort between teammates and technology. 14:45 Ballroom BC Session Database How Netflix Ensures Highly-Reliable Online Stateful Systems Joseph Lynch Principal Software Engineer @Netflix Building Highly-Reliable and High-Leverage Infrastructure Across Stateless and Stateful Services Under most stateless services are stateful databases, caches, and systems which form the bedrock applications are built on. 15:55 Ballroom BC Session Resiliency Orchestrating Resilience: Building Modern Asynchronous Systems Sai Pragna Etikyala Technical Lead @Twilio Building asynchronous, event-driven systems can be daunting. Managing states, ensuring resilience, maintaining traceability, and handling a myriad of other challenges often require more effort than building the functionality itself. 17:05 Seacliff D Unconference Unconference: Designing for Resilience What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.