Magic Pocket is used to store all of Dropbox’s data. It is a horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, is able to maintain 99.99% availability and has extremely high durability guarantees, while being more cost efficient than operating in the cloud.

This system is able to facilitate new drive technology, handle millions of queries per second, and automatically identify and repair hundreds of hardware failures per day. We are constantly innovating in this space and work closely with hard drive vendors to adopt the latest drive technology (https://techcrunch.com/2020/10/26/dropbox-begins-shift-to-high-efficiency-western-digital-shingled-magnetic-recording-disks/). Each storage device contains 100+ drives and is multiple petabytes in size. Given the blast radius of single device failures, it is critical that our erasure codes and traffic are all built with this in mind.

In this talk we will deep dive into the architecture of Magic Pocket, some early key design patterns that we still live by to this day, and the challenges of operating such a system at this scale in order to be cost efficient and support many critical requirements.

The key takeaways for this talk are:

• Provide an overview of the architecture of Magic Pocket. This includes key services, databases, how multi-region replication works, repairs, and a discussion on the storage devices.
• Key architecture lessons, which had the most impact on Magic Pocket.
• How we are able to operate such a system, while being extremely cost efficient.

Our system is much cheaper than operating in the cloud, but it operates with a high bar. We discuss these challenges in more detail for others looking to make this transition and what these trade-offs look like.