Industry and academia need each other. Far from the fires of production, researchers have the time to ask the big questions. Sometimes they get lucky, but detached from real world constraints they risk irrelevance by inventing and solving imaginary problems. Industry has the customers, the data, and the problems that only come with scale. They want answers to the big questions, but fear the risks of a bad investment. Despite this deep independence, collaborations between industry and academia are rare.

We present our experience: a fruitful industry/academic collaboration. We describe how a “big idea” -- lineage-driven fault injection -- evolved from a theoretical model into an automated failure testing service at Netflix. This collaboration required us to take risks, to accept defeats, and to constantly evolve our approach to “make it work”. We sketch the architecture of the automated failure testing service while providing intuition for why it works. Along the way, we will describe the challenges (expected as well as unexpected, technical as well as ideological) that arose, and how we overcame them.