Everyone talks about model quality. Fewer people talk about what happens when that model has to serve real traffic without falling over, or why the system ended up shaped the way it did in the first place.

At Netflix, our ML serving platform runs in two modes. Synchronous serving answers requests in real time, for use cases such as recommendations and search. Asynchronous serving processes predictions in large batches, often on a schedule, for workloads such as Automatic Speech Recognition and Machine Translation. Both modes serve similar models, but they fail in different ways, and some of that traces back to Conway's Law: how ML practitioners and the teams consuming their models are organized and shaped the systems we built.

We'll share some of the incidents we've run into across both modes, what we learned, and what we changed as a result. We'll also talk about our experience rebuilding this platform from the ground up and turning it into a shared offering that other teams at Netflix now build on. Along the way, we'll cover topics such as admission control, concurrency limits, and what's worth measuring in production. This talk is meant to be a practical, honest look at building ML infrastructure, including people and org structure behind it.

Main Takeaways:

1. Lessons on architecting for both sync and async inference, and where the two diverge.
2. How to build concurrency controls that hold up in production.
3. How to measure and observe inference systems correctly.
4. How to debug production inference failures at scale.
5. How org structure shapes serving architecture, for better or worse.