Online Controlled Experiments: Introduction, Insights, Scaling, and Humbling Statistics
Track: Continuous DeliveryLocation:Grand Ballroom B/CAbstract:
The web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations).
From front-end user-interface changes to backend algorithms, online controlled experiments are now utilized to make data-driven decisions at Amazon, eBay, Facebook, Google, Intuit, LinkedIn, Microsoft, Netflix, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, running online controlled experiments at scale—hundreds of concurrent experiments on a given day at Bing—has taught us many lessons.
You will learn about controlled experiments through an introduction and real examples. We will share key insights, cultural challenges, scaling challenges, and humbling statistics.