Accelerating Netflix Data: A Cross-Team Journey from Offline to Online

Abstract

At Netflix, certain use cases demand the rapid transfer of massive datasets—such as 50 TB—from offline to online systems. Doing this efficiently, without disrupting applications interacting with our online systems, presents a significant challenge. Traditional data transfer methods, such as using batch processing systems and loading data into online systems via PUT APIs, posed significant scalability and cost hurdles, often leading to performance bottlenecks and impacting system efficiency. To overcome these limitations, an innovative architectural solution was developed. This approach involved transforming offline data into an optimized format through pre-processing, staging this data, transforming offline data into RocksDB SST file format, staging these files in the cloud, and enabling direct, on-demand ingestion into the serving system.

This process necessitated navigating complex internal discussions and aligning diverse stakeholders on new technical strategies. It also required rapidly adapting initial prototypes to address urgent customer needs by initially prioritizing speed to onboard them to the prototype, before shifting efforts towards building a robust, scalable production system. Crucially, cross-functional collaboration proved essential. Teams from various domains worked closely to define requirements, overcome challenges, and ensure seamless implementation.

Ultimately, this collaborative effort led to the successful deployment of a system that provides enhanced performance, reducing data deployment time by 99% (from days to just 30 minutes) and cutting costs by 70%. This presentation will delve into the journey of transforming data pipelines at scale, highlighting the key technical strategies, strategic decisions, and crucial team efforts that made this significant improvement possible.

Key Takeaways:

  • Discover the challenges and solutions for large-scale data movement from batch storage to online serving systems.
  • Understand the innovative architectural approach to improve data deployment efficiency.
  • Strategic decision-making and problem-solving in a high-pressure environment.
  • The importance of cross-functional team collaboration in solving complex engineering problems.

Date

Tuesday Nov 18 / 05:05PM PST ( 50 minutes )

Location

Ballroom A

Share

From the same track

Session

Modernizing Relevance at Scale: LinkedIn’s Migration Journey to Serve Billions of Users

Tuesday Nov 18 / 01:35PM PST

How do you deliver relevant and personalized recommendations to nearly a billion professionals—instantly, reliably, and at scale? At LinkedIn, the answer has been a multi-year journey of architectural reinvention.

Speaker image - Nishant Lakshmikanth

Nishant Lakshmikanth

Engineering Manager @LinkedIn, Leading Infrastructure for "People You May Know" and "People Follows", Previously @AWS and @Cisco

Session

Monolith Down: Cleaning Up After the Great Identity Migration Disaster

Tuesday Nov 18 / 10:35AM PST

One does not simply migrate a monolith. Imagine a team working on a monolith-to-microservices migration of a healthcare portal. A foundational first step - migrating to a commercial identity provider - takes 9 months, only to bring the entire portal crashing down on release day.

Speaker image - Sonya Natanzon

Sonya Natanzon

VP of Engineering @Heartflow, Decomplexifier, Software Architect, Healthcare and Life Sciences Specialist, and International speaker

Session

Migrating from Monolith to Modern SaaS: Untangling 40 Years of Legacy

Tuesday Nov 18 / 11:45AM PST

What does it take to successfully modernize and migrate a mission-critical line of business running on a 40-year-old monolith to a more modern SaaS?

Speaker image - Indu Alagarsamy

Indu Alagarsamy

Principal Engineer @CircleCI, 25 Years of Software Development Experience

Session

Unconference: Navigating Major Architecture Migrations

Tuesday Nov 18 / 03:55PM PST

Session

Migrating Uber Eats Feeds to Webview

Tuesday Nov 18 / 02:45PM PST

Uber Eats has many surfaces developed using native-first design. Historically these were built on the Android and iOS stacks. To accelerate development and enable rapid iteration and experimentation, while preserving the native-first design, a webview-powered stack was developed.