In this talk, Erin Doyle, will walk through her experience with a critical 3-day-long incident. She will discuss how, as a Staff Engineer, she missed a key opportunity to help prevent the incident and how Staff+ Engineers can influence a culture that can prevent similar situations. She’ll then talk about the lessons learned from how Staff+ Engineers can help improve incident management as well as the post-mortem process. Lastly, she’ll touch on the automated solution she led implementation of in order to prevent these kinds of problems going forward.
This story-telling will show you:
- Ways that Staff+ Engineers can help foster a culture that could have prevented such an incident.
- Lessons learned on how technical leaders can more effectively manage incidents.
- Tips for how to encourage more useful incident post-mortems.
- Examples of how to take a pragmatic approach to implementing improvements after an incident.
Interview:
What is the focus of your work these days?
I had been a full-stack developer for 20+ years and then at the beginning of 2023 I moved to a Platform team. So my work now is in the Platform/DevOps/SRE space. I especially have enjoyed working on projects that improve developers' workflow and experience, CI/CD, automation, and collaborating with Developers and setting up the infrastructure they need for their solutions.
What's the motivation for your talk?
My experience with this 3-day incident really opened my eyes to how much opportunity there is for Staff+ Engineers to positively affect the outcomes of incidents (or even prevent them altogether). You can have tools and automation to make the process a little smoother but they don't help much if you don't have the right processes and culture in place around the tools. When tensions and stakes are high, as humans, we can feel pressure, rush, cut corners, miss important things; we don't think clearly and aren't always at our best. Having level-headed leaders with experience to drive and direct through the processes and safe-guards you put in place can make the difference. And mindful leaders can plant the seeds and provide the needed care and feeding to foster a culture that can cut down on a number of contributing factors to incident root causes.
How would you describe the persona and level of the target audience?
The target audience for this talk includes all current or aspiring technical leaders. While the track is Staff+ I believe this talk can apply to Senior Engineers and up, as well as Engineering Managers. Whether you already have an incident management progress in place, have no process at all, or are somewhere in between, I hope you will find useful tips and perspectives in this talk.
What do you want this persona to walk away with from your presentation?
I hope attendees will walk away from this talk at the very least introspecting on the current culture and incident management process at their company. I hope this talk will give them ideas for how they can influence and improve their company's culture or find places where their process can be improved or they can drive change around incidents.
What do you think is the next big disruption in software?
I really want to say something other than "AI" but I really do think we're at the cusp of something really big with AI and Machine Learning. There are minds brainstorming all across the world on what we can do with this technology and I think we're going to see applications beyond our imaginations soon. What I'm really interested in though is the application to the developer workflow. We already have a bunch of tools and techniques popping up using AI to enhance developers' tasks. It will be exciting to see what new tools and applications come out, what becomes part of the typical developer's quiver, and how these tools and techniques elevate developer productivity to allow for new levels of achievement and discovery.
Speaker
Erin Doyle
Sr. Staff Software Engineer @Lob, with 20+ Years Previously as a Full Stack Engineer and Instructor @Egghead
Erin Doyle is a Sr. Staff Software Engineer @Lob. For the last 20+ years prior she’s been working as a Full Stack Engineer with a focus on the Front-End. She’s also an instructor for Egghead.io and given talks and workshops focusing on building the best and most accessible experiences for users and developers.