Interview with Mohit vora & Haley Tucker

QCon: What is your role today?

Mohit: I am responsible for what we internally refer to as the Control Plane of our Content Delivery Network. In classical networking speak, the Control Plane refers to the part of the architecture responsible for making routing decisions. So that is something that my team works on. We decide where each playback sessions streams from based on a bunch of things. Besides that, we also make predictions on content popularity and proactively cache that content in opportune locations throughout the network.

Haley: When a customer clicks play, I’m responsible for providing them with the best possible playback experience for their device and user preferences, while still enforcing contract and business policies. I own and operate the systems that I will talk about which means I’m doing everything from requirements and coding for new features to production on-call triage.

QCon: Mohit, what is Open Connect?

Mohit: Open Connect is Netflix’s purpose-built CDN. It’s a directed caching CDN. Together with the Control Plane it is comprised of 1000’s of cache nodes and networking equipment located in ISP and Internet Exchange data centers.

QCon: You mentioned Open Connect is a directed caching CDN. What do you mean by Directed Caching?

Mohit: When you can predict what data is going to be accessed more often vs others, you can proactively direct the data to appropriate tiers of your caching infrastructure in advance. This contrasts to moving data around based on actual cache hits / misses.

QCon: Some might think what you plan to discuss are really Netflix-only scale problems. Are these things others could benefit from and use in their architectures?

Haley:Yes, I think so. We are focusing on lessons that can be applied regardless of scale, and I have definitely worked on systems in the past that could have benefited from similar changes.

QCon: What about Directed Caching? That’s a pretty specialized use case.

Mohit: With the popularity metric of our catalog, we are able to pre-position content close to our subscribers using Direct Caching, and hence reduce network costs for Internet Service Providers. We are able to achieve no cache churn in the serving state (i.e. no reactive caching). Most high-scale distributed architectures that utilize caching enjoy a high amount of predictable access patterns (think something like cache pre-warming). Directed caching is applicable to them.

QCon: Haley, how does Netflix “manage the playback experience”?

Haley: We set up the context for the player which allows it to stream. This context contains the video, audio, and text assets that are displayed to the customer in the format that the device needs. It also contains recommendations for which audio and subtitles to start playback with based on our previous signals. In addition to setting up the context, these microservices are engaged for each phase of the lifecycle of a session (license, session events, bookmarks, etc.).

QCon: What types of things will you discuss in your talk that relate to managing the playback experience?

Haley: We use rules engines for customized viewing experiences and leverage configuration deployments.

For example, rules are used for things like filtering (remove streams or bad encodes that have been identified to cause problems for a particular device) and to enable/disable features at a fine-grained level for devices or different UI’s that may or may not support them.

Our rules are all deployed via a pub-sub mechanism to all services running our code. There are times, for example, when we receive a call that a device is having issues with a particular language track. We can put a filter in place and after about 5 minutes that filters out that language track for just that subset of devices having problems.

Additionally, one of the changes we recently made to our architecture is to allow clients to control their own protocols. So, instead of a one-sized-fits all API, client protocols live in groovy scripts which are deployed in a manner similar to configuration files. This allows TVs to integrate with us in a manner different from iPhones and they can emit data back to the client in the format that they want it.

QCon: Can you give me an idea of some of the principles and patterns we will hear about in your talk?

Haley: We are focused on lessons learned for this talk, so we would like people to come away with an understanding of the problems we faced and how we addressed them.

Mohit: Some of the things you will hear include:

• Trend toward microservices, configuration, and improved flexibility
• Challenges around consistent hashing
• Employing a control loop feedback system for load balancing
• Our simplistic model for predicting content popularity over time