Speaker
Abstract
In this talk, Erin Doyle, will walk through her experience with a critical 3-day-long incident. She will discuss how, as a Staff Engineer, she missed a key opportunity to help prevent the incident and how Staff+ Engineers can influence a culture that can prevent similar situations. She’ll then talk about the lessons learned from how Staff+ Engineers can help improve incident management as well as the post-mortem process. Lastly, she’ll touch on the automated solution she led implementation of in order to prevent these kinds of problems going forward.
This story-telling will show you:
- Ways that Staff+ Engineers can help foster a culture that could have prevented such an incident.
- Lessons learned on how technical leaders can more effectively manage incidents.
- Tips for how to encourage more useful incident post-mortems.
- Examples of how to take a pragmatic approach to implementing improvements after an incident.
Interview
I had been a full-stack developer for 20+ years and then at the beginning of 2023 I moved to a Platform team. So my work now is in the Platform/DevOps/SRE space. I especially have enjoyed working on projects that improve developers' workflow and experience, CI/CD, automation, and collaborating with Developers and setting up the infrastructure they need for their solutions.
My experience with this 3-day incident really opened my eyes to how much opportunity there is for Staff+ Engineers to positively affect the outcomes of incidents (or even prevent them altogether). You can have tools and automation to make the process a little smoother but they don't help much if you don't have the right processes and culture in place around the tools. When tensions and stakes are high, as humans, we can feel pressure, rush, cut corners, miss important things; we don't think clearly and aren't always at our best. Having level-headed leaders with experience to drive and direct through the processes and safe-guards you put in place can make the difference. And mindful leaders can plant the seeds and provide the needed care and feeding to foster a culture that can cut down on a number of contributing factors to incident root causes.
The target audience for this talk includes all current or aspiring technical leaders. While the track is Staff+ I believe this talk can apply to Senior Engineers and up, as well as Engineering Managers. Whether you already have an incident management progress in place, have no process at all, or are somewhere in between, I hope you will find useful tips and perspectives in this talk.
I hope attendees will walk away from this talk at the very least introspecting on the current culture and incident management process at their company. I hope this talk will give them ideas for how they can influence and improve their company's culture or find places where their process can be improved or they can drive change around incidents.
I really want to say something other than "AI" but I really do think we're at the cusp of something really big with AI and Machine Learning. There are minds brainstorming all across the world on what we can do with this technology and I think we're going to see applications beyond our imaginations soon. What I'm really interested in though is the application to the developer workflow. We already have a bunch of tools and techniques popping up using AI to enhance developers' tasks. It will be exciting to see what new tools and applications come out, what becomes part of the typical developer's quiver, and how these tools and techniques elevate developer productivity to allow for new levels of achievement and discovery.
Topics
QCon San Francisco 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Staff+ Engineering Skills Hosted by Krys Flores Staff Software Engineer @Crunchyroll, Host of O'Reilly's The Staff Engineer's Career RoadmapFrom the same track
Monday 2 October
10:35 Seacliff ABC Session Staff Plus Engineering An Incident Story: Tips for How Staff+ Engineers Can Impact Incidents Erin Doyle Founding Engineer @Quotient, 20+ Years Across Full Stack Development in Web and Mobile, and Platform Engineering In this talk, Erin Doyle, will walk through her experience with a critical 3-day-long incident. 11:45 Ballroom A Session Staff Plus Engineering Things Every Staff+ Engineer Should Know Joy Ebertz Principal Engineer @Imprint, Blogger, and Speaker, Previously @Harness, @Split, & @Box As staff+ engineers, we're often thrown into the deep end and expected to navigate huge amounts of ambiguity including ambiguity about what our jobs even are. It's common to feel like there's a huge amount of trial and error or even luck. 13:35 Seacliff ABC Session Staff Plus Engineering Managing Staff+ Engineers: Opportunities and Challenges Adam Schirmacher Staff Engineer & Manager of Staff Engineers @Gusto Staff+ engineers can be a powerful force in your organization…if you let them. Effectively managingStaff+ engineers requires different strategies than many managers are used to employing. 14:45 Pacific DEKJ Session Panel: Staff+ Engineering Skills Erin Doyle, Adam Schirmacher, Joy Ebertz, Caleb Hyde Staff+ engineering is a critical role in any high-performing engineering organization. But what does it take to get promoted or get hired into a staff role? What does it take to keep it? Join us for a panel discussion with experienced Staff+ engineers who will share their insights. 15:55 Ballroom BC Session Staff Plus Engineering Risk and Failure on the Path to Staff Engineer Caleb Hyde Site Reliability Engineer @Expel Even as inaction can be a risk, deciding which actions to take in one's career involves choosing between different bets. I've developed a rubric for judging specific risks that I've taken. 17:05 Seacliff D Unconference Unconference: Staff+ Engineering What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.