Warning message

  • The service having id "twitter" is missing, reactivate its module or save again the list of services.
  • The service having id "facebook" is missing, reactivate its module or save again the list of services.
  • The service having id "google_plus" is missing, reactivate its module or save again the list of services.
  • The service having id "linkedin" is missing, reactivate its module or save again the list of services.

Presentation: Architecting Distributed Databases for Failure



1:40pm - 2:30pm

Key Takeaways

  • Failure happens... design distributed system to handle failure gracefully
  • Hear lessons learned building a distributed NoSQL Database
  • Learn principles and architectural patterns when building a distributed database.


Running distributed systems in production can be tremendously challenging. In this session, we will cover common problems and failures seen with distributed systems, and discuss design patterns that can used to maintain data integrity and availability when everything goes wrong. We will use Druid as a real world case study of how these patterns are implemented in an open source technology.

Attendees will learn first hand about the multitude of software, hardware, network, and data center problems that can arise with running distributed systems, and the features that are required for availability and survivability. To provide real world examples, we will examine the architecture of Druid, and how the system is designed to power applications that need to be up 24/7. We will also cover common pitfalls with running distributed systems in various environments, including the trade-offs with on-premise and cloud deployments.

Finally, we will cover best practices around properly instrumenting monitoring and alerting for distributed systems. We will examine various open source technologies that can be used for efficient monitoring, and how these technologies can be used to maintain the availability of your cluster.

We hope this session will help you better use, design, and monitor your data systems.

Additional Links:

  • Intro reading list about distributed systems: 


Interview with Fangjin Yang

QCon: Your talk is called "Architecting Distributed Databases for Failure." Can you tell me about it?

Fangjin: The talk is about architectural patterns that exist with building distributed databases and some of the lessons learned building an open source distributed database. All distributed databases have a lot of principles in common (such as the way that distributed systems can survive failure). Academia has been talking about this for ages. What I want to do is talk about some of these main principles and use examples from an open source project called Druid as an example of a real world case study in which these principles are implemented.

QCon: You mention Druid as an example of building a distributed database, what is Druid?

Fangjin: Druid is an open source column-oriented distributed database. It's primarily designed for analytics and business intelligence. If you're familiar with data warehousing world, it's a database that's really good for OLAP queries and streaming your data into the database directly.

QCon: So how will your background developing Druid affect your talk?

Fangjin: One thing I was hoping to do was talk a little bit about how you do a rolling update on a distributed system, and why it makes sense to have a management component be a part of rolling updates.

From the very beginning with Druid, we decided that we wanted to be able to do rolling deployments. Druid has various components built in to allow you to do take down on process at a time without any loss of data. If you bring that process back up, Druid has a very quick way of restoring state. There is a management piece that says "hey, one of my servers went down, I might need to start redistributing load and replicating missing data." If the server that went down comes back pretty fast, then the management piece says "OK, the server has been restored I don't have to do anything." It's part of the reason why Druid has many different components. It has dedicated components monitoring the state of the cluster. I would leverage similar stories of these types of experiences in my talk.

QCon: How is your talk going to be structured?

Fangjin: First, I thought I would discuss about the different types of failures that can occur (from the small failures to the oh crap, everything is broken failures). From there, I want to cover a bit about the main principles that distributed systems implement to handle these various classes of failures and the typical patterns that systems follow. The third part would be more of the case study where I've implements some of these principles like the rolling deployment example about. The last part is really about how how does all this actually function in a working system.

QCon: What is the main take away someone might have coming to your talk?

Fangjin: I think the main takeaway that I hope people get out of this talk is things are going to break. It's not the end of the world. There are a lot of things you can do to survive various types of failures.

Similar Talks

Co-creator of Apache Kafka, Co-founder & Head of Engineering @Confluent
VP of Global Platform and Infrastructure @PayPal
Director, Head of Data Science and Engineering @Jawbone
CTO and co-founder @AzulSystems
Engineering Manager and Technical Lead for Real Time Analytics @Twitter


Covering innovative topics

Monday Nov 16

Tuesday Nov 17

Wednesday Nov 18

Conference for Professional Software Developers