Nick Kallen, Twitter Engineer; FlockDb co-creator

Track: Architectures you've always wondered about

Abstract: One of the biggest opportunities for innovation on the web today is mining the huge amount of data from Twitter and similar platforms. Unfortunately, supporting these datasets is labor-intensive due to immature storage solutions. We will discuss Twitter's experience with its four fundamental data types and query patterns: tweets, timelines, social graphs, and search indices. For each of these, Twitter implemented custom datastores because existing solutions were insufficient. Based on that experience, we will propose requirements for a general-purpose distributed database that would foster broad innovation on the web.