Interview with Fangjin Yang

QCon: Your talk is called "Architecting Distributed Databases for Failure." Can you tell me about it?

Fangjin: The talk is about architectural patterns that exist with building distributed databases and some of the lessons learned building an open source distributed database. All distributed databases have a lot of principles in common (such as the way that distributed systems can survive failure). Academia has been talking about this for ages. What I want to do is talk about some of these main principles and use examples from an open source project called Druid as an example of a real world case study in which these principles are implemented.

QCon: You mention Druid as an example of building a distributed database, what is Druid?

Fangjin: Druid is an open source column-oriented distributed database. It's primarily designed for analytics and business intelligence. If you're familiar with data warehousing world, it's a database that's really good for OLAP queries and streaming your data into the database directly.

QCon: So how will your background developing Druid affect your talk?

Fangjin: One thing I was hoping to do was talk a little bit about how you do a rolling update on a distributed system, and why it makes sense to have a management component be a part of rolling updates.

From the very beginning with Druid, we decided that we wanted to be able to do rolling deployments. Druid has various components built in to allow you to do take down on process at a time without any loss of data. If you bring that process back up, Druid has a very quick way of restoring state. There is a management piece that says "hey, one of my servers went down, I might need to start redistributing load and replicating missing data." If the server that went down comes back pretty fast, then the management piece says "OK, the server has been restored I don't have to do anything." It's part of the reason why Druid has many different components. It has dedicated components monitoring the state of the cluster. I would leverage similar stories of these types of experiences in my talk.

QCon: How is your talk going to be structured?

Fangjin: First, I thought I would discuss about the different types of failures that can occur (from the small failures to the oh crap, everything is broken failures). From there, I want to cover a bit about the main principles that distributed systems implement to handle these various classes of failures and the typical patterns that systems follow. The third part would be more of the case study where I've implements some of these principles like the rolling deployment example about. The last part is really about how how does all this actually function in a working system.

QCon: What is the main take away someone might have coming to your talk?

Fangjin: I think the main takeaway that I hope people get out of this talk is things are going to break. It's not the end of the world. There are a lot of things you can do to survive various types of failures.