Designing systems that take failure into account from the start is hard. Sometimes it’s very tempting to take shortcuts that will adversely affect your system in the long run. However, there are steps one can take to avoid these shortcuts and build a fault-tolerant system.

This talk is a case study in transitioning Guitar, an internal integration testing framework, to Spanner. Spanner is a database developed internally at Google that provides a fast, distributed data store for applications in addition to distributed transactions, replication, and automated backups. Previously Guitar relied upon a hodge-podge of Bigtables, in-memory data structures, and custom mechanisms for recovering state upon failure. This previous architecture resulted in a system that was unscalable, unreliable, and an impediment to developing new features that were sorely needed.

We will first go into details of the previous architecture and how the mechanisms in place for dealing with failure were inadequate and not properly thought out. We will then discuss the transition plan from the old to the new system, which could be generally applicable to other migrations, and the lessons we learned for how to deal with failure properly. In addition, we will discuss how we transitioned our heavily used production system for all of our clients with no downtime and in a fully controlled and gradual manner.