Speaker
Abstract
As engineers, we’re used to managing failure, but long-running outages hit differently. They stretch teams, systems, and assumptions about how incidents “should” play out. In this talk, we’ll dive into real examples of incidents that dragged on far longer than anyone expected, and unpack what they revealed about our systems, processes, and mental models.
We’ll explore what these situations taught us about coordination under pressure, shifting system behavior, and the limitations of our current practices for detection and response. We will also look at how a mindset of curiosity helped us make sense of the mess — not just to resolve the immediate situation, but to improve how we adapt, learn, and build stronger systems and teams.
Topics
QCon San Francisco 2025 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Wednesday 19 November
10:35 Seacliff ABC Session Incidents The Human Toll of Incidents & Ways To Mitigate It Kyle Lexmond Production Engineer @Meta, Previously @AWS and @Twitter Have you ever wondered what it's like to respond to a significant incident? Walk through an hour by hour reconstruction of an incident response or two, focusing on what it was like to be "in the room" and the human response to the incidents. 11:45 Seacliff ABC Session Incidents When Incidents Refuse to End Vanessa Huerta Granda Resilience Engineering Manager @Enova, Co-Author of the Howie Guide on Post Incident Analysis, Board Member for the Resilience in Software Foundation As engineers, we’re used to managing failure, but long-running outages hit differently. They stretch teams, systems, and assumptions about how incidents “should” play out. 13:35 Seacliff ABC Session Staff Plus Engineering The Ironies of A^2 I^2 J. Paul Reed Staff Incident Operations Manager @Chime In this talk, we'll explore some of the "ironies" of automation—and now, artificial intelligence—in their interactions with software operators (i.e. you), especially during high consequence, high tempo situations (aka incidents). 14:45 Seacliff ABC Session Incident Response Week-Long Outage: Lifelong Lessons Molly Struve Staff Site Reliability Engineer @Netflix Routine database upgrades should be straightforward, especially with familiar, well-established technology. We were confident heading into our Elasticsearch upgrade, equipped with a solid plan and excited to see performance gains like we had seen from past upgrades. 15:55 Seacliff ABC Session Incident Analysis The Time it Wasn't DNS Sean Klein Principal Technical Program Manager - Modern Incident Analysis @Microsoft Azure In January of 2023, the Microsoft Azure Wide Area Network experienced a global outage. If you were a Microsoft customer at the time, you were impacted by this outage.