An Incident Story: Tips for How Staff+ Engineers Can Impact Incidents

QCon San Francisco 2023

Session Staff Plus Engineering

An Incident Story: Tips for How Staff+ Engineers Can Impact Incidents

Monday Oct 2 / 10:35AM PDT, Seacliff ABC

Abstract

In this talk, Erin Doyle, will walk through her experience with a critical 3-day-long incident. She will discuss how, as a Staff Engineer, she missed a key opportunity to help prevent the incident and how Staff+ Engineers can influence a culture that can prevent similar situations.  She’ll then talk about the lessons learned from how Staff+ Engineers can help improve incident management as well as the post-mortem process.  Lastly, she’ll touch on the automated solution she led implementation of in order to prevent these kinds of problems going forward.

This story-telling will show you:

  • Ways that Staff+ Engineers can help foster a culture that could have prevented such an incident.
  • Lessons learned on how technical leaders can more effectively manage incidents.
  • Tips for how to encourage more useful incident post-mortems.
  • Examples of how to take a pragmatic approach to implementing improvements after an incident.

Interview

I had been a full-stack developer for 20+ years and then at the beginning of 2023 I moved to a Platform team.  So my work now is in the Platform/DevOps/SRE space.  I especially have enjoyed working on projects that improve developers' workflow and experience, CI/CD, automation, and collaborating with Developers and setting up the infrastructure they need for their solutions.

My experience with this 3-day incident really opened my eyes to how much opportunity there is for Staff+ Engineers to positively affect the outcomes of incidents (or even prevent them altogether).  You can have tools and automation to make the process a little smoother but they don't help much if you don't have the right processes and culture in place around the tools.  When tensions and stakes are high, as humans, we can feel pressure, rush, cut corners, miss important things; we don't think clearly and aren't always at our best.  Having level-headed leaders with experience to drive and direct through the processes and safe-guards you put in place can make the difference.  And mindful leaders can plant the seeds and provide the needed care and feeding to foster a culture that can cut down on a number of contributing factors to incident root causes.

The target audience for this talk includes all current or aspiring technical leaders.  While the track is Staff+ I believe this talk can apply to Senior Engineers and up, as well as Engineering Managers.  Whether you already have an incident management progress in place, have no process at all, or are somewhere in between, I hope you will find useful tips and perspectives in this talk.

I hope attendees will walk away from this talk at the very least introspecting on the current culture and incident management process at their company.  I hope this talk will give them ideas for how they can influence and improve their company's culture or find places where their process can be improved or they can drive change around incidents.

I really want to say something other than "AI" but I really do think we're at the cusp of something really big with AI and Machine Learning.  There are minds brainstorming all across the world on what we can do with this technology and I think we're going to see applications beyond our imaginations soon.  What I'm really interested in though is the application to the developer workflow.  We already have a bunch of tools and techniques popping up using AI to enhance developers' tasks.  It will be exciting to see what new tools and applications come out, what becomes part of the typical developer's quiver, and how these tools and techniques elevate developer productivity to allow for new levels of achievement and discovery.

Topics

Staff Plus Engineering Incident Management Platform Engineering
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon San Francisco 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Monday 2 October

10:35 Seacliff ABC Session Staff Plus Engineering An Incident Story: Tips for How Staff+ Engineers Can Impact Incidents Erin Doyle Founding Engineer @Quotient, 20+ Years Across Full Stack Development in Web and Mobile, and Platform Engineering 11:45 Ballroom A Session Staff Plus Engineering Things Every Staff+ Engineer Should Know Joy Ebertz Principal Engineer @Imprint, Blogger, and Speaker, Previously @Harness, @Split, & @Box 13:35 Seacliff ABC Session Staff Plus Engineering Managing Staff+ Engineers: Opportunities and Challenges Adam Schirmacher Staff Engineer & Manager of Staff Engineers @Gusto 14:45 Pacific DEKJ Session Panel: Staff+ Engineering Skills Erin Doyle, Adam Schirmacher, Joy Ebertz, Caleb Hyde 15:55 Ballroom BC Session Staff Plus Engineering Risk and Failure on the Path to Staff Engineer Caleb Hyde Site Reliability Engineer @Expel 17:05 Seacliff D Unconference Unconference: Staff+ Engineering