MLOps: A Framework to Build Reliable ML Systems [Half-Day]

Over the last decade Machine Learning has become ubiquitous, powering applications across a variety of domains - from web search to autonomous drones. Under the hood, getting these ML systems to production and ensuring good performance over time remains challenging.

MLOps is an emerging discipline bringing together best practices from ML, DevOps and Data Engineering to tackle the challenges of building effective real-world ML systems: bootstrapping data for new ML projects, improving data & label quality, model evaluation, model deployment, ML observability and more.

This workshop will introduce you to important concepts and developments in MLOps, with a combination of lectures and hands-on projects. The goal is to help bridge the gap between state-of-the-art ML modeling, and building real-world ML systems.

Key Takeaways

1 The production ML lifecycle (project scoping → data collection → modeling →  deployment → observability). 

2 Data collection for real-world ML applications: strategies for acquiring and labeling data, with attention to label quality and bias.

3 ML model evaluation: designing evaluation to proxy real-world performance, importance of dynamic benchmarks and behavioral tests.

4 ML model deployment: Common approaches & tradeoffs

5 Observability in ML systems: system and performance metrics, delayed feedback, data & concept drift, building closed-loop feedback systems.


Nihit Desai

CTO and co-founder @RefuelAI, Previously staff engineer @Facebook, recommender systems @Instagram & search quality @LinkedIn

Nihit Desai is the CTO and co-founder of Refuel.AI, building ML infrastructure for teams working with unstructured data. Prior to this, he was a staff engineer at Facebook, building and scaling their human-in-the-loop ML efforts for content integrity. In prior roles, he has worked on recommender systems at Instagram, and on search quality at LinkedIn. He is the co-author of the MLOpsRoundup newsletter, and the creator of the co:rise MLOps course.

Read more
Find Nihit Desai at:


Thursday Oct 27 / 09:00AM PDT ( 3 hours )


Bayview A


Level intermediate


Machine Learning Observability



  • Participants should have knowledge of introductory machine learning concepts
  • Participants should be familiar with programming in Python. Knowledge of libraries commonly used for data and ML workflows such as numpy, pandas and sklearn is especially helpful (but not required).
  • Participants should bring their own laptops (Linux or MacOS operating systems preferred), with Python 3.6+ and Docker pre-installed.