The primary Google code base receives up to 20+ code changes per minute with 50% of the source files changing every month.  Most products are developed and released from ‘head’ and released on weekly or even daily schedules.  To keep things working while we iterate quickly, we rely heavily on automated testing - running millions of tests each day.

To make this possible, we have built a large-scale continuous testing service utilizing our cloud computing infrastructure.  The scale - the change rate, the number of tests and the size of the code base - restricts the types of test selection and prioritization schemes that can be applied. In addition, the system is used to both quickly and accurately identify changes that cause test failures and to identify potential release candidates where all tests pass, which can be competing goals in terms of test prioritization.  In this talk, I’ll describe the basic architecture of the test automation system and the supporting build system infrastructure.  I’ll then describe the coarse-grained dependency analysis being used to select tests and how build system optimizations can compensate for imprecise dependency analysis.   I’ll also discuss how the usage of the system has changed and how we are currently evolving the test prioritization scheme.