Online Controlled Experiments: Introduction, Insights, Scaling, and Humbling Statistics

Location:

Grand Ballroom B/C

Track:

Time:

Monday, 10:30am - 11:20am

Abstract:

The web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations).

From front-end user-interface changes to backend algorithms, online controlled experiments are now utilized to make data-driven decisions at Amazon, eBay, Facebook, Google, Intuit, LinkedIn, Microsoft, Netflix, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, running online controlled experiments at scale—hundreds of concurrent experiments on a given day at Bing—has taught us many lessons.

You will learn about controlled experiments through an introduction and real examples. We will share key insights, cultural challenges, scaling challenges, and humbling statistics.

Ronny Kohavi

Ronny Kohavi is a partner architect in the Online Services Division at Microsoft. He joined Microsoft in 2005 and founded the Experimentation Platform team in 2006. He was previously the director of data mining and personalization at Amazon.com, and the Vice President of Business Intelligence at Blue Martini Software, which went public in 2000, and later acquired by Red Prairie. Prior to joining Blue Martini, Kohavi managed MineSet project, Silicon Graphics' award-winning product for data mining and visualization. He joined Silicon Graphics after getting a Ph.D. in Machine Learning from Stanford University, where he led the MLC++ project, the Machine Learning library in C++ used in MineSet and at Blue Martini Software. Kohavi received his BA from the Technion, Israel. He was the General Chair for KDD 2004, co-chair of KDD 99's industrial track with Jim Gray, and co-chair of the KDD Cup 2000. He was an invited speaker at the National Academy of Engineering in 2000, a keynote speaker at PAKDD 2001, an invited speaker at KDD 2001, EC 10 (2010), Recsys 2012, emetrics 2013, and SIGIR. His papers have over 20,000 citations.