Speaker
Abstract
Even in the ethereal world of software, everything has a limit. And, once that limit is reached, very bad things can happen. In this talk, we will explore the problem of saturation: when a software system runs into one of these limits. We'll see a sampling of the many different kinds of system overload, how our systems are always potentially at risk of saturation no matter how well designed they are, and what we can do to prepare for the day when a system in our charge has been pushed beyond what it can normally handle.
Interview
My session is about *saturation*, an extremely common failure mode in cloud-based software systems. If senior developers are more explicitly aware of saturation, they can design their systems to both identify when the risk of saturation is increasing, and more quickly recover when they encounter it.
In the new age of LLM-driven-development, we're changing our systems more quickly. This has the effect of increasing the load on our software delivery tooling, which means we can expect saturation-related problems to increase with the volume of the changes.
Reliability work tends to focus on the prevention of incidents that occurred previously. A significant challenge is: how do I design my system so I can recover more quickly from an incident that hasn't happened before?
Tooling that makes it easier for an operator to manually reduce load in different parts of their system during an incident.
The consistently high quality of the speakers, who all have deep technical experience.
Topics
$2,995, Conference (3 days). Current pricing ends November 10th. All pass options.
QCon San Francisco 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Monday 16 November
10:35 Seacliff ABC Session Resiliency The Freeze Paradox: Why Stopping Deployments Doesn't Stop Failures Prachi Jain, Sandhya Narayan When systems fail at Netflix and in an environment this complex, they sometimes will, the instinctive response is to freeze deployments. 11:45 Seacliff ABC Session Resilience Adapt or Drift: Resilience Engineering When AI Moves the Operating Point Andrew Hatch Engineering Leader and SRE Manager @Cisco ThousandEyes, With 25+ Years Building Software, Operations, SRE, and Platform Teams Across Australia, India, and the United States Resilient systems do not stay resilient by standing still. They survive by adapting. But adaptation comes with risk: under sustained pressure, organizations and systems can slowly drift toward failure while still appearing to operate normally. 13:35 Seacliff D Unconference Unconference: Resilience Engineering 14:45 Seacliff ABC Session Reliability Saturation: How Your Software Will Fail at Scale Lorin Hochstein Staff Software Engineer @Airbnb, Writes @surfingcomplexity.blog, Previously @Netflix and Member of the Resilience in Software Foundation Even in the ethereal world of software, everything has a limit. And, once that limit is reached, very bad things can happen. In this talk, we will explore the problem of saturation: when a software system runs into one of these limits. 15:55 Seacliff ABC Session Incidents What Is Incident Severity, but a Lie Agreed Upon? Em Ruppe Technical Incident Commander @Chime, Previously @SendGrid and @Twilio, and Product and Training @Jeli.io Is your incident severity a complex word problem? Do you arbitrate which severity an incident should be during the incident itself? Do you know what the difference between your severities are? Do you even have incident severity in your org? 17:05 Seacliff ABC Session It's Never the Thing That Broke: Learning From Incidents at (AI) Speed Vanessa Huerta Granda Resilience Engineering Manager @Enova, Co-Author of the Howie Guide on Post Incident Analysis, Board Member for the Resilience in Software Foundation When something breaks, our first instinct is to focus on the thing that failed. But in complex systems, the failure is usually the result of many things going right, changing, and interacting in ways we didn't anticipate.