Elliot: I am a software engineer at Netflix on the Personalization Infrastructure team. We develop data pipelines and other infrastructure to help the personalization algorithms team make recommendations.

Elliot: A significant part of the real time data we process is made up of what people browse through on Netflix and what people watch, and people are watching Netflix at all hours during the day. All of this data comes in through Kafka, so everything is fairly real time. There obviously is a lot of data cleaning that needs to be done, and depending on the algorithms that consume the data, we have to adjust our filtering criteria accordingly.

Elliot: Trending Now is one of the first use cases at Netflix where we try to make use of real time data for personalization. We are also one of the first at Netflix to be using Spark Streaming. There have been many questions about how well this will work, especially in a production environment at Netflix's scale. Our team had looked into Spark Streaming maybe a year or two ago and it wasn’t quite mature enough at the time. Many improvements have been made since then, which has prompted us to revisit it. We wanted to share our learnings as we build this system and continue to improve it.

Elliot: A bit of code will probably come up in a few places about Spark Streaming specifically, but that should only be a small part of the discussion. I'll focus on different patterns we tried.