Presentation: Mantis: Netflix's Event Stream Processing System
Netflix customers stream over two billion hours of content each month, accounting for over a third of downstream Internet traffic during peak hours. At this scale, Netflix's systems generate and collect millions of events every second, such as request traces, streaming client activities, and system metrics. It is essential for engineers to process such data streams efficiently and reliably to support real-time monitoring and alerting, outlier detection, application diagnostics, trend prediction, and many other operations.
Mantis is a new stream processing framework in development at Netflix. It provides users with the capabilities to write scalable stream processing jobs without having to worry about hard problems such as managing continuous data flow in a distributed environment or ensuring fault tolerance.
This talk will discuss how Mantis is being used at Netflix, the unique features of Mantis, and how Mantis is implemented. In particular:
- The programming model of Mantis. Mantis allows users to specify a directed acyclic graph to process streaming data. On each computation node, Mantis allows users to specify the processing logic using a rich set of reactive programming APIs.
- The support of auto scaling. Mantis is able to scale out each computation node based on the system load. This makes each Mantis job elastic to user-defined metrics.
- The optimization on stream localities and multiple stream sources. Mantis allows its jobs to tap into various type of streams available in Netflix, and schedules the jobs so they are as close to the stream data as possible.
Tracks
Covering innovative topics
Monday, 3 November
-
Architectures You've Always Wondered about
The newest and biggest Internet architectures
-
Real World Functional
Putting functional programming concepts to work in the real world.
-
The Future of Mobile
The future of mobile and performance improvements
-
Continuous Delivery: From Heroics to Becoming Invisible
Continuous Delivery philosophies, cultures, hiccups, and best practices.
-
Unleashing the Power of Streaming Data
This track explores a variety of use-cases, platforms, and techniques for processing and analyzing stream data from the companies deploying them at scale!
-
Sponsored Solutions Track I
Tuesday, 4 November
-
Engineering for Product Success
Architectures that make products more successful
-
Reactive Service Architecture
Reactive, Responsive, Fault Tolerant and More.
-
Modern CS In the Real World
How modern CS tackles problems in the real world.
-
Applied Machine Learning and Data Science
Understand your big big data!
-
Deploying at Scale
Containerizing Applications, Discovering Services, and Deploying to the Grid.
-
Sponsored Solutions Track II
Wednesday, 5 November
-
Beyond Hadoop
Emerging Big Data Frameworks and Technology
-
Scalable Microservice Architectures
This track addresses the ways companies with hundreds of fine-grained web-services (e.g. Netflix, LinkedIn) manage complexity!
-
Java at the Cutting Edge
The latest and greatest in the Java ecosystem
-
Engineering culture
Successes and failures in creating an engineering culture.
-
Next gen HTML5 and JS
How Web Components, the Future of CSS, and more are changing the web.
-
Sponsored Solutions Track III