How do you know what 55 millions users like? Wix.com is conducting over 1000 experiments per month on production to understand which features our users like and which hurt or improve our business.

In this talk we’ll explain how our engineering team is supporting our product managers in making the right decisions and getting our product roadmap on the right path. We will also present some of the open source tools we developed that help us experimenting our products on humans.

While A/B test is a very known and familiar methodology for conducting experiments on production when you do that on a large scale by changing your system behavior every 9 minutes, it entails many challenges in the organization level from developers, product managers, QA, marketing and management.

In this talk we will explain what is the lifecycle of an experiment, some of the challenges we faced and the effect on our development process and product evolution.

Some of the topics we’ll discuss are:

• How a feature experiment begins its life
• How do you let non technical product managers control the experiment while preventing mistakes
• How do you make A/B testing an integral part of your product methodology
• How an experiment go live, what is the lifecycle of an experiment from beginning to end
• What do your product managers and engineering have to be careful about when introducing new feature
• Unpleasant failures we had while introducing new features
• How can support help users when every user may be part of different experiment
• How can we find if an experiment is causing errors when you have millions of permutations [at least 2^(number of active experiments)]
• How does having A/B testing constantly running affects your product roadmap and the challenges it entails on your system’s architecture

At Wix we have developed our 3rd generation experiment system called PETRI, which we open sourced, that helps us maintain some order in a chaotic system that keep changing. We will also explain how PETRI works, what are the patterns in conducting experiments that will have a minimal effect on performance and user experience.