Conference: Nov 13-15, 2017
Workshops: Nov 16-17, 2017
Presentation: Scaling Dropbox
Duration
Level:
- Intermediate
Persona:
- Architect
Key Takeaways
- Learn the challenges Dropbox faces with reads and writes, and how they address them.
- Understand the key design principle behind the architecture of a large storage application.
- Learn lessons and tips addressing massive scale, cascading failure, architecture, MySql, MemCached, and more.
Abstract
Dropbox is a technology company that builds simple, powerful products for people and businesses. We’ve grown enormously since launching in 2008, surpassing 500 million signups and storing over 500 petabytes of user data. Since we started, Dropbox users have created 3.3 billion connections by sharing with each other and are saving 1.2 billion files per day. In this talk, we’ll discuss how Dropbox’s infrastructure evolved over the years and how it looks today as well the challenges and lessons learned on the way.
Interview
Preslav: I’m software engineer on the storage team at Dropbox. In the storage team, we’re responsible for storing files in Dropbox (which is critical for us). Durability is very important for Dropbox because it’s a service for backing up your files. It’s a very different problem. That’s why we’re so concerned about durability. We have built internal software (Magic Pocket) with that in mind.
Before joining this team, I spend a year in infrastructure performance.
Preslav: We store more than 500 petabytes.
Preslav: Some parts of it are literally a replacement for S3. Because we use it for internal purposes, we don’t need to build everything that S3 offers. Magic Pocket gives us the flexibility to tune and optimize for our use cases at Dropbox.
Preslav:I’ll be focusing on the whole Dropbox architecture. There is already a good talk on the internet from early 2012 for how Dropbox scaled as startup from 0 to 50 million users so I will focus on how we scaled as a more mature company from 50 to 500 million. What’s unique about Dropbox's architecture. What are the scaling issues we faced and how we solved them.
Unlike many companies, we really care about consistency. If you delete a file in your folder, we need to have those in strict order. Another thing that is different for Dropbox is we serve a lot of writes. Most companies- if you think of YouTube, Facebook- they serve some number of writes. But really it is 10 or 100 or 1000 more reads than writes. For us, reads and writes are roughly equal. So that makes our scaling challenges a bit different than you might have heard about before.
Preslav: One example is with our memcache (we modified the library for our use case). Usually you use a memcache by just reading from it. If the cache is not available, you read from the database. But for Dropbox, memecache is also on the write path. Everything you need to write to the database, you need to write to memcache in order to keep it strictly consistent. I’ll tell you why this is a problem: when your memcache is not available, then your database is also not available.
Preslav: Because of the high number of writes, we needed to introduce sharding much earlier than other companies. While sharding is simple concept in theory, in practice it is a little bit more challenging. We’ll talk about historical things that didn’t go perfectly, like shard isolation and collocation.
Then, we will discuss why couldn’t scale as organization using mysql directly and how we design systems in Dropbox today knowing failure is inevitable.
Similar Talks
.
Tracks
Monday Nov 7
-
Architectures You've Always Wondered About
You know the names. Now learn lessons from their architectures
-
Distributed Systems War Stories
“A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.” - Lamport.
-
Containers Everywhere
State of the art in Container deployment, management, scheduling
-
Art of Relevancy and Recommendations
Lessons on the adoption of practical, real-world machine learning practices. AI & Deep learning explored.
-
Next Generation Web Standards, Frameworks, and Techniques
JavaScript, HTML5, WASM, and more... innovations targetting the browser
-
Optimize You
Keeping life in balance is a challenge. Learn lifehacks, tips, & techniques for success.
Tuesday Nov 8
-
Next Generation Microservices
What will microservices look like in 3 years? What if we could start over?
-
Java: Are You Ready for This?
Real world lessons & prepping for JDK9. Reactive code in Java today, Performance/Optimization, Where Unsafe is heading, & JVM compile interface.
-
Big Data Meets the Cloud
Overviews and lessons learned from companies that have implemented their Big Data use-cases in the Cloud
-
Evolving DevOps
Lessons/stories on optimizing the deployment pipeline
-
Software Engineering Softskills
Great engineers do more than code. Learn their secrets and level up.
-
Modern CS in the Real World
Applied, practical, & real-world dive into industry adoption of modern CS ideas
Wednesday Nov 9
-
Architecting for Failure
Your system will fail. Take control before it takes you with it.
-
Stream Processing
Stream Processing, Near-Real Time Processing
-
Bare Metal Performance
Native languages, kernel bypass, tooling - make the most of your hardware
-
Culture as a Differentiator
The why and how for building successful engineering cultures
-
//TODO: Security <-- fix this
Building security from the start. Stories, lessons, and innovations advancing the field of software security.
-
UX Reimagined
Bots, virtual reality, voice, and new thought processes around design. The track explores the current art of the possible in UX and lessons from early adoption.
Conference for Professional Software Developers
