80-90% of data science is data cleaning and feature engineering. However, if we were to plot a count of what all the data science tools are for, we would find that most innovation happens in data infrastructure and modeling. We want to change that and make data scientists much more productive while also improving the quality of their work.

In this talk I will describe the machine learning platform we wrote on top of spark to modularize these steps. This allows easy reuse of components, simplifying model building and changes. The framework simplifies the data preparation and feature building stages with reusable classes for each data source, making subsequent feature generation a matter of a few lines of code.

Model selection classes wrap the functionality of pre-existing and custom algorithms to provide a uniform interface for modeling, allowing rapid iteration in model evolution. By breaking the machine learning process into a series of simple to implement and interchangeable pieces we have democratized the process of building machine learning models.