Speaker
Abstract
Teams at NASA and JPL that create mission-critical software for spacecraft take a principled approach to fault tolerance. Let's see how those same principles, centered around a concept of transparency, can help us achieve reliability in pragmatic, modern software delivery settings.
As our society continues to depend more and more heavily on software, the need for that software to be reliable also increases. At the same time, the software systems we build as practitioners continue to become more and more complex, with many moving parts and unexpected, emergent behavior. How can those systems become as reliable and robust as the Voyager 2 deep space probe that made news recently and is still in service 46 years after its launch?
In this talk, we'll be drawing inspiration from an architectural paper that captures experiences from the aerospace industry. "GN&C [Guidance, Navigation, and Control] Fault Protection Fundamentals" by Robert D. Rasmussen describes four key principles for building fault-tolerant software, involving transparency of (1) objectives; (2) models; (3) knowledge; and (4) control. For each principle, we'll identify use cases from well-known software or protocols (e.g. HTTP) where we can see the principle in action; or, if the principle is *not* commonly applied, we'll describe how it might look in familiar settings like invoking RESTful or GRPC APIs.
Interview
I'm currently working on broad architectural refactoring projects to help update parts of our architecture to gain more engineering leverage (identifying and extracting common components for reuse, for example).
I'm passionate about fault tolerance and reliability, and have been for a long time. I will admit to enjoying putting a service under rigorous load or stress testing, seeing it fall over, and then fixing it so it no longer falls over that way! But in general, I see a great opportunity for the software industry to continue updating our practices to build more of that reliability and resilience in by design. There are groups, like NASA, who have been focused on fault tolerant software for a long time: after all, a small mistake could turn a deep space probe into a $100M brick. They've developed a lot of rigor and techniques to be successful at this, and the rest of the industry can probably learn a thing or two!
This talk is probably most useful to software architects or tech leads, as we will be discussing techniques and patterns that an organization would want to apply broadly to gain the benefits. That said, the talk should be broadly accessible to a wider engineering audience, as we'll be explaining the principles in the talk as we go, and illustrating them with examples.
I'd primarily hope that attendees gain a new way of thinking about fault tolerance and reliability; being able to learn about a new point of view on a familiar problem is often the most valuable takeaway from talks.
Topics
QCon San Francisco 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Designing for Resilience Hosted by Javier Fernandez-Ivern Staff Software Engineer @Netflix with Over 24 Years in Software EngineeringFrom the same track
Tuesday 3 October
10:35 Ballroom BC Session Architecture Disaster Recovery Across a Million Pieces Michelle Brush Engineering Director, SRE @Google, Previously Director of HealtheIntent Architecture @Cerner Corporation & Lead Engineer @Garmin, Author of "2 out of the 97 Things Every SRE Should Know" Data recovery is more than just backing up and restoring a data store. The goal of any disaster recovery effort is getting the system back to working as expected across all of its parts. 11:45 Ballroom BC Session Reliability Designing Fault-Tolerant Software with Control System Transparency Jon Moore Staff Software Engineer @Stripe with over 35 years of software engineering experience across both academia and industry Teams at NASA and JPL that create mission-critical software for spacecraft take a principled approach to fault tolerance. Let's see how those same principles, centered around a concept of transparency, can help us achieve reliability in pragmatic, modern software delivery settings. 13:35 Ballroom BC Session Resiliency How Do We Talk to Each Other? How Surfacing Communication Patterns in Organizations Can Help You Understand and Improve Your Resilience Nora Jones Founder and CEO @jeli_io, Founder of Learning From Incidents (LFI) Online Community and Conference As a system increases in inevitable complexity, it becomes impossible for a single operator to have a clear, unambiguous understanding of what's happening in the system. Understanding the system requires a joint effort between teammates and technology. 14:45 Ballroom BC Session Database How Netflix Ensures Highly-Reliable Online Stateful Systems Joseph Lynch Principal Software Engineer @Netflix Building Highly-Reliable and High-Leverage Infrastructure Across Stateless and Stateful Services Under most stateless services are stateful databases, caches, and systems which form the bedrock applications are built on. 15:55 Ballroom BC Session Resiliency Orchestrating Resilience: Building Modern Asynchronous Systems Sai Pragna Etikyala Technical Lead @Twilio Building asynchronous, event-driven systems can be daunting. Managing states, ensuring resilience, maintaining traceability, and handling a myriad of other challenges often require more effort than building the functionality itself. 17:05 Seacliff D Unconference Unconference: Designing for Resilience What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.