Speaker
Abstract
People used to ask me all the time how to figure out if their chaos test has “passed,” and I’d always say “well, that’s a loaded question.” To confirm that a chaos test “passed,” we need to do verification of hypotheses - sometimes you’re trying to prove some system behavior occurred in response to a stimulus, while other times you’re trying to prove the absence of a change in system behavior. Take this already nebulous concept, and now think about making it generic enough that the core validation logic can be re-used by any engineer running any kind of experiment on any one of our products. Then, try to do all of this in a complex distributed technical environment where it’s hard enough just to determine whether an application was healthy in the first place! That’s exactly the problem that the chaos engineering team at Vanguard has been tackling with the recent addition of automated assertions to the internal chaos tooling. In this talk, you’ll learn about when it’s appropriate to define “pass” and “fail” for a chaos experiment, and when it might not be, and you’ll get to take a peek under the hood at the way that Vanguard engineers are automatically verifying their hypotheses in the context of chaos experiments.
Interview
I actually just started a new role now operating in more of an architect capacity than where I previously was very narrowly scoped, Site Reliability Engineering. So focusing on architecture that supports all of our developer experience platform and enablement of software engineering excellence across our entire I.T. organization.
In this talk I hope to build on some of what I've shared about Vanguard's chaos engineering strategy in some prior talks, and talk a little bit about the level of maturity that we've reached now where we're not just experimenting in an exploratory way - but using the results of our Chaos experiments to make some assertions about the reliability of our systems and hopefully make sure that others understand how to do the same with their Chaos experiments as well.
I think that any technician will take a lot away from this talk, especially anyone who works in large enterprises, because that's the environment that I'm working in at Vanguard. Anyone who has some experience running chaos experiments in the past or has an interest in running chaos experiments in their organizations. So anyone with a site reliability engineering background, or just some experience with chaos experimentation will really enjoy this talk.
When they walk away from the presentation they'll certainly have a feel for the architecture and what we've built if they want to do something similar in their own organizations. But I don't expect that that's what most will take away. I hope primarily that people will walk away with the idea to put some assertions around some of their chaos experiments, also to do some exploratory testing without assertions and to determine when is the right time to do each of those things.
Topics
QCon San Francisco 2022 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Effective SRE Hosted by Casey Rosenthal CEO, Co-Founder @verica_ioFrom the same track
Wednesday 26 October
10:35 Bayview Session SRE The Endgame of SRE Amy Tobey Senior Principal Engineer and SRE practice Leader @Equinix The containers are deployed and the builds are green. Yaml flows through the system, linted, reviewed, tested, and shipped with ease and regularity. Our intrepid SRE finds themself at a crossroads. The infrastructure is great but teams still struggle to maintain error budgets. 11:50 Bayview Session SRE Did the Chaos Test Pass? Christina Yakomin Senior Site Reliability Engineering Specialist @Vanguard_Group People used to ask me all the time how to figure out if their chaos test has “passed,” and I’d always say “well, that’s a loaded question.” To confirm that a chaos test “passed,” we need to do verification of hypotheses - sometimes you’re trying to prove some system behavior occurred in response… 13:40 Seacliff ABC Session [Panel] SRE: Is it Working? Courtney Nash, Amy Tobey, Christina Yakomin, Sasha Rosenbaum How does SRE mature from a craft with a wide range of skills and levels of expertise to a mature discipline? 14:55 Bayview Session SRE Rethinking Reliability: What You Can (and Can't) Learn From Incidents Courtney Nash Co-founder @The VOID, Previously @Verica, @Holloway, @Fastly, @O’Reilly Media, @Microsoft, & @Amazon This talk presents research collected from the VOID—an open database of public incident reports. Containing over 2,000 reports for almost 700 organizations, the database allows for more structured review and research about software-related incident reporting. 16:10 Bayview Session SRE The Eternal Sunshine of the Toil-Less Prod Sasha Rosenbaum Director of the Cloud Services Black Belt Team @RedHat One of the most important decisions in building an SRE practice is what kind of work should be assigned to the SRE team, and in what percentages.