Presentation: Experimenting on Humans
How do you know what 55 millions users like? Wix.com is conducting over 1000 experiments per month on production to understand which features our users like and which hurt or improve our business.
In this talk we’ll explain how our engineering team is supporting our product managers in making the right decisions and getting our product roadmap on the right path. We will also present some of the open source tools we developed that help us experimenting our products on humans.
While A/B test is a very known and familiar methodology for conducting experiments on production when you do that on a large scale by changing your system behavior every 9 minutes, it entails many challenges in the organization level from developers, product managers, QA, marketing and management.
In this talk we will explain what is the lifecycle of an experiment, some of the challenges we faced and the effect on our development process and product evolution.
Some of the topics we’ll discuss are:
- How a feature experiment begins its life
- How do you let non technical product managers control the experiment while preventing mistakes
- How do you make A/B testing an integral part of your product methodology
- How an experiment go live, what is the lifecycle of an experiment from beginning to end
- What do your product managers and engineering have to be careful about when introducing new feature
- Unpleasant failures we had while introducing new features
- How can support help users when every user may be part of different experiment
- How can we find if an experiment is causing errors when you have millions of permutations [at least 2^(number of active experiments)]
- How does having A/B testing constantly running affects your product roadmap and the challenges it entails on your system’s architecture
At Wix we have developed our 3rd generation experiment system called PETRI, which we open sourced, that helps us maintain some order in a chaotic system that keep changing. We will also explain how PETRI works, what are the patterns in conducting experiments that will have a minimal effect on performance and user experience.
Tracks
Covering innovative topics
Monday, 3 November
-
Architectures You've Always Wondered about
The newest and biggest Internet architectures
-
Real World Functional
Putting functional programming concepts to work in the real world.
-
The Future of Mobile
The future of mobile and performance improvements
-
Continuous Delivery: From Heroics to Becoming Invisible
Continuous Delivery philosophies, cultures, hiccups, and best practices.
-
Unleashing the Power of Streaming Data
This track explores a variety of use-cases, platforms, and techniques for processing and analyzing stream data from the companies deploying them at scale!
-
Sponsored Solutions Track I
Tuesday, 4 November
-
Engineering for Product Success
Architectures that make products more successful
-
Reactive Service Architecture
Reactive, Responsive, Fault Tolerant and More.
-
Modern CS In the Real World
How modern CS tackles problems in the real world.
-
Applied Machine Learning and Data Science
Understand your big big data!
-
Deploying at Scale
Containerizing Applications, Discovering Services, and Deploying to the Grid.
-
Sponsored Solutions Track II
Wednesday, 5 November
-
Beyond Hadoop
Emerging Big Data Frameworks and Technology
-
Scalable Microservice Architectures
This track addresses the ways companies with hundreds of fine-grained web-services (e.g. Netflix, LinkedIn) manage complexity!
-
Java at the Cutting Edge
The latest and greatest in the Java ecosystem
-
Engineering culture
Successes and failures in creating an engineering culture.
-
Next gen HTML5 and JS
How Web Components, the Future of CSS, and more are changing the web.
-
Sponsored Solutions Track III