Netflix runs one of the largest big data analytics infrastructure in the public cloud. Our platform leverages the scalability, reliability, and flexibility of the cloud to move quickly and innovate.

In this talk, we will discuss the overall big data platform architecture and dive into the two key design choices that underpin our platform: Storage and Orchestration. We will discuss how we leverage S3 as our data warehouse storage layer. We rely on Parquet as our primary storage format and will cover the advantages of using Parquet on S3 along with many of the features and optimizations provided by this advanced file format,

We will also discuss our open source federated job management and orchestration layer, Genie... Every day Netflix runs tens of thousands of jobs across the numerous heterogeneous (Hadoop, Presto, Spark, etc.) clusters. From Spark and Pig ETL SLA jobs to ad-hoc interactive queries on Presto to data movement with Sqoop or indexing with Druid, Genie is used to orchestrate this diverse set of use cases across multiple clusters in our environment. Genie also helps us manage clusters and job lifecycles in the cloud.

Finally, we will cover where we plan on taking Genie including scaling job resources via Docker, and more.