Presentation: Real Time Recommendations using Spark Streaming


2:55pm - 3:45pm



Key Takeaways

  • Understand the architecture, concerns, and challenges building a high volume real-time streaming solution like Trending Now leveraging Spark Streaming. 
  • Hear practical advice implementing a real-time streaming service, such as joining multiple live streams of data, data persistence patterns, and resiliency considerations.
  • Learn some of the strengths and weakness Netflix uncovered using Spark Streaming to better understand whether it fits your individual use case.


Recommendations play a vital role in a great Netflix experience. Traditionally, these recommendations are precomputed using viewing history, scroll activity, and a variety of other signals in a near-line fashion. To be able to react more quickly to surges and dips in interest, we introduced the Trending Now row that makes use of real time data as an additional signal for generating recommendations. This allows us to not only personalize this row based on the context like time of day and day of week, but also react to sudden changes in collective interests of members, due to real-world events.

We will discuss the data pipeline that we built to process Netflix user activities in real time for the Trending Now row.  We will share our experiences with developing, monitoring, and productionalizing a system that uses Kafka, Spark Streaming, and Cassandra.


QCon: What is your role today?

Elliot: I am a software engineer at Netflix on the Personalization Infrastructure team. We develop data pipelines and other infrastructure to help the personalization algorithms team make recommendations.

QCon: What is the scale of recommendations at Netflix? It must be huge.

Elliot: Definitely. With over 70 million of subscribers, there is a lot of traffic every day. In addition to streaming content, we also collect telemetry about how our users are engaging with Netflix. Since we want to generate the best recommendations and user experience, we have to consider all of that data.

QCon: How much of that data is used for real time? Is it just a small subset or massive amounts of data?

Elliot: A significant part of the real time data we process is made up of what people browse through on Netflix and what people watch, and people are watching Netflix at all hours during the day. All of this data comes in through Kafka, so everything is fairly real time. There obviously is a lot of data cleaning that needs to be done, and depending on the algorithms that consume the data, we have to adjust our filtering criteria accordingly.

QCon: How do you describe the persona of the target audience of this talk?

Elliot: Someone who is considering using Spark Streaming or using it more extensively. They would also be involved in implementation.

QCon: What’s the motivation for your talk?

Elliot: Trending Now is one of the first use cases at Netflix where we try to make use of real time data for personalization. We are also one of the first at Netflix to be using Spark Streaming. There have been many questions about how well this will work, especially in a production environment at Netflix's scale. Our team had looked into Spark Streaming maybe a year or two ago and it wasn’t quite mature enough at the time. Many improvements have been made since then, which has prompted us to revisit it. We wanted to share our learnings as we build this system and continue to improve it.

QCon: Can you give an example? When you say share your learnings, can you tell me about one of the stories, one of the lessons you’ll tell?

Elliot: A couple things I plan to discuss are challenges we faced around joining multiple input streams and some Spark Streaming quirks. We have also done quite a bit on the operations side, including things like failure handling, calculating metrics, and monitoring. These are critical pieces that make the system resilient and production ready.

QCon: How deep do you get into the weeds with the implementation details?

Elliot: A bit of code will probably come up in a few places about Spark Streaming specifically, but that should only be a small part of the discussion. I'll focus on different patterns we tried.

QCon: Can you give me an example of something actionable that a senior dev/lead might gain from coming to you talk?

Elliot: They should come away with an idea about how we used the technologies we chose for building a system like this, and whether they would be suitable for their use cases. Hopefully, they will also find some of our experiences with Spark Streaming useful.

Speaker: Elliot Chow

Senior Software Engineer @Netflix

Elliot is a software engineer at Netflix on the Personalization Infrastructure team. He graduated from UC Berkeley (B.S.) and Stanford (M.S.) and has previously worked at eBay and Apple. Currently, he builds big data systems using a variety of technologies including Scala, Spark (Streaming), Kafka, and Cassandra.

Find Elliot Chow at



Monday Nov 7

Tuesday Nov 8

Wednesday Nov 9

Conference for Professional Software Developers