Presentation: In-memory caching: curb tail latency with Pelikan


10:35am - 11:25am


Key Takeaways

  • Gain insight into the first principles used by Twitter in building Pelikan
  • Understand design decisions and architectural patterns used in the implementation of Twitter’s replacement for Redis and Memecachd.
  • Learn about techniques used to improve latency in high throughput systems like Twitter.


In-memory caching in a distributed setting is ubiquitous, of which users usually expect both high throughput and low latencies. Meanwhile, to actually achieve them requires understanding the details and subtleties in low-level system behavior, including memory management, threading model, locking, network I/O, etc.

Pelikan is a framework to implement distributed caches such as Memcached and Redis. This talk discusses the system aspects that are important to the performance, especially tail latencies, of such services. It also covers the decisions we made in Pelikan, and how they help us reason about performance even before we run benchmarks.


QCon: What is Pelikan?

Yao: Pelikan is the in memory caching framework that I have been working on to replace the caching use cases at Twitter. 

People know that Twitter is, historically, a very large Memcached shop. It is also a very large Redis shop, and we have done our own forks based on those both Memcached and Redis. As we accumulate more experience with these existing caching services, we have a good idea of how to do it well. We realized that instead of having two solutions working toward the same purpose, we can unify them in a single architecture that is optimized for the use cases Twitter needs. 

In short, Pelikan is the replacement for Memcached and Redis and everything they are responsible for at Twitter.

QCon: So is it kind of a ground up replacement for cache tools like Memcached and Redis?

Yao: I would describe it as two fold. From an architectural (or design) point of view, it is a clean slate design. We come at it from first principles asking questions like ‘What is the problem we are trying to solve using caching in memory storage?’ and ‘What are the requirements?’ If we pose this problem as a new question today, how are we going to design an architecture that works for the scale and works for the datacenter that we have or that is available on the market? So from a design point of view, it is a clean slate. 

However, new code is usually bad code (or if not bad, it’s ugly code). There is significant risk in using new coding in a critical piece of the architecture. In terms of implementation, we are not so strict. We actually prefer using existing code if it fits into the design well.  So let’s say you have an existing networking library that works really well, there is no need to write your own library to handle connections. You can just copy the code that handles connections and drop that into your design. You can do that, because it serves exactly the same purpose. With implementation, we are much more practical and utilitarian in examining solutions that we have, including Memcached and Twemcached (which is our fork Redis). We will take a look at whoever has a good library out there that does something similar. gRPC is another one. There are actually a lot of code that does more or less the same thing, so we would just look at all the codebases we can find. If there is something that we can use, we will use it, and we will give credit to open source project.

In reality, what happened is we started by copying code and writing new code about one to one. We were re-using about 50% of our code from existing open source projects and I lost track of the ratio over time because then we started re-factoring and polishing the code. But we started with 50% re-use. The new code ratio has gone up slightly since then, because, as we add more features and as we re-write, we tend to bring more code of our own into the project.

QCon: How would you describe target audience core persona?

Yao: I have two groups of people in mind. The first being architects and designers who are designing or re-designing systems from the ground up. Considering these aspects will help them avoid some pitfalls that they will regret later. The second group being SRE’s (or people who actually need to operate these systems). This group organically showed up after I gave other similar talks.  They are the ones getting the question of "why is my PIII high?" If the design is flawed (or if you run in a very unstable environment and share resources with other processes), a lot of things are beyond your control. These are the people who will benefit if the service, especially the quality of service, can be guaranteed more strongly or can be reasoned about more easily.

QCon: What’s the goal of your talk?

Yao: Since this talk is in the performance track, the goal of my talk is to focus on the performance aspect of Pelikan. I will focus on latency because there are services that deliver very high throughput but, if you look at their latency (especially tail latencies), there tend to be outliers. I believe Redis sometimes has this problem. Pretty much all questions, discussions, troubleshooting sessions are about why is something slow unexpectedly? So that’s really what I want to talk about.

Speaker: Yao Yue

Distributed Systems Engineer Working on Cache @Twitter

Distributed Systems Engineer Working on Cache at Twitter

Find Yao Yue at

Similar Talks

Tech Lead for Timelines Infrastructure Team / Sr Staff Software Engineer @Twitter
Core Systems Libraries Software Engineer @Twitter
Observability Software Engineer @Twitter
Tech Lead of Manhattan Team @Twitter
Provisioning Engineering SE @Twitter
Senior Software Engineer, Playback Features @Netflix



Monday Nov 7

Tuesday Nov 8

Wednesday Nov 9

Conference for Professional Software Developers