Abstract
Honeycomb is the state of the art in observability: customers send us lots of data and then compose complex, ad-hoc queries. Most are simple, some are not. Some are REALLY not; this load is both complex, spontaneous, and urgent. It would be prohibitively expensive to size a server cluster to handle these big queries quickly, so we took a different approach: farm the work out to Lambda, Amazon's serverless offering.
In this model, Lambda becomes an on-demand accelerator for our always-on servers. The benefits are immense, improving response times by an order of magnitude. But the challenges are numerous and often unexpected. In this talk, I'll review the benefits (user experience on demand!) and constraints (everything in AWS has a limit!) of serverless-as-accelerator, and give practical advice based on our own hard-won experience.
Topics
QCon San Francisco 2022 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Architectures You've Always Wondered About Hosted by Randy Shoup SVP Engineering @Thrive Market, Previously @eBay, @Google, @Stitch FixFrom the same track
Wednesday 26 October
10:35 Ballroom A Session Architecture Amazon DynamoDB: Evolution of a Hyper-Scale Cloud Database Service Akshat Vig Distinguished Engineer @MongoDB, Previously Senior Principal Engineer NoSQL@AWS Amazon DynamoDB is a cloud database service that provides consistent performance at any scale. Hundreds of thousands of customers rely on DynamoDB for its fundamental properties: consistent performance, availability, durability, and a fully managed serverless experience. 11:50 Ballroom A Session Architecture Honeycomb: How We Used Serverless to Speed Up Our Servers Jessica Kerr Principal Developer Evangelist @honeycombio Honeycomb is the state of the art in observability: customers send us lots of data and then compose complex, ad-hoc queries. Most are simple, some are not. Some are REALLY not; this load is both complex, spontaneous, and urgent. 13:40 Ballroom A Session Architecture Azure Cosmos DB: Low Latency and High Availability at Planet Scale Mei-Chin Tsai, Vinod Sridharan Azure Cosmos DB is a fully-managed, multi-tenant, distributed, shared-nothing, horizontally scalable database that provides planet-scale capabilities and multi-model APIs for Apache Cassandra, MongoDB, Gremlin, Tables, and the Core (SQL) APIs. 14:55 Ballroom A Session Architecture From Zero to A Hundred Billion: Building Scalable Real Time Event Processing At DoorDash Allen Wang Software Engineer @DoorDash, previously Lead for real-time data infrastructure team @Netflix At DoorDash, real time events are an important data source to gain insight into our business but building a system capable of handling billions of real time events is challenging. 16:10 Ballroom A Session Architecture Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System Facundo Agriel Software Engineer / Tech Lead @Dropbox, previously @Amazon Magic Pocket is used to store all of Dropbox’s data.